The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.
Difference between revisions of "Funtoo:Metatools/Advanced Usage"
Line 128: | Line 128: | ||
Now that merge-scripts is cloned, we will need to create a {{c|/root/.merge}} configuration file. Use the following file as a starting point: | Now that merge-scripts is cloned, we will need to create a {{c|/root/.merge}} configuration file. Use the following file as a starting point: | ||
{{file| | {{file|name=/root/.merge|body= | ||
[sources] | [sources] | ||
Line 163: | Line 163: | ||
The {{c|[work]}} section is used to define paths where the merge-scripts will do their work. {{c|merge-scripts}} should point to the path where you cloned your merge-scripts repository, as {{c|merge-all-kits.py}} will use this location to look for package-set definitions. These are files that define which catpkgs go in which kits. The {{c|source}} and {{c|destination}} settings above are good defaults, and define where the merge-scripts will clone source repositories and destination (written to) repositories, such as kits and meta-repo. These are kept in two separate hierarchies so they don't get mixed up. | The {{c|[work]}} section is used to define paths where the merge-scripts will do their work. {{c|merge-scripts}} should point to the path where you cloned your merge-scripts repository, as {{c|merge-all-kits.py}} will use this location to look for package-set definitions. These are files that define which catpkgs go in which kits. The {{c|source}} and {{c|destination}} settings above are good defaults, and define where the merge-scripts will clone source repositories and destination (written to) repositories, such as kits and meta-repo. These are kept in two separate hierarchies so they don't get mixed up. | ||
== Generating New Kits == | |||
With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be stored on '''repohost'''. Here are the steps you'd perform: | |||
{{console|body= | |||
# ##i##cd /root | |||
# ##i##python3 merge-all-kits.py push | |||
}} | |||
{{c|merge-all-kits.py}} will proceed to create new kits and meta-repo, and will push them up to repos@repohost:wildrepo/staging/meta-repo, repos@repohost:wildrepo/staging/core-kit, etc. This process can take quite a while but has been optimized to run quickly on multi-core systems. | |||
== Using New Kits == | |||
Now that the new meta-repo and kits are created, here's how you'll use them on an existing Funtoo system, instead of your official Funtoo meta-repo and kits. First, we'll want to modify {{c|/etc/ego.conf}} as follows: | |||
{{file|name=/etc/ego.conf|body= | |||
[global] | |||
sync_user = root | |||
sync_base_url = repos@repohost:wildrepo/staging/{repo} | |||
# You can have whatever [kits] section you want, below... | |||
}} | |||
You will want to make sure that whatever system is connecting to '''repohost''' is permitted by gitolite, and has its {{c|/root/.ssh/id_rsa.pub file}} stored and committed to the gitolite-admin repository's {{c|keydir/}} and is referenced in {{c|conf/gitolite.conf}}. Otherwise, gitolite will not allow this system to connect. Of course, if you are configuring this on the system that's running {{c|merge-all-kits.py}}, it already has RW permissions to these repositories. Otherwise, you will want to make sure that gitolite has the system's keys as part of its {{c|@reporead}} group. | |||
{{Note|We change the {{c|sync_user}} to {{c|root}} to force {{c|ego sync}} to use {{c|/root/.ssh/id_rsa}} for authentication. By default, {{c|ego sync}} will attempt to use the {{c|portage}} user which does not have a private key installed, and thus will not be able to authenticate with gitolite.}} | |||
Now, let's move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run {{c|ego sync}}: | |||
{{console|body= | |||
# ##i##cd /var/git | |||
# ##i##mv meta-repo meta-repo.official | |||
# ##i##ego sync | |||
}} | |||
Ego will now sync your custom repository. If you type {{c|emerge -auDN @world}}, ego will now be using your custom kits, rather than the official Funtoo ones :) |
Revision as of 21:52, May 20, 2018
This page documents how to set up a local development environment that will allow full local testing of changes. This includes generating your own meta-repo and kits with your custom changes, as well as getting metro set up for unit tests.
Overview
For our local development setup, we will be using gitolite on our development workstation. Gitolite will make things quite a bit easier by managing git repositories for us. Think of it as your own private GitHub that has no Web user interface (we modify its settings by pushing to its special gitolite-admin
repo) and you'll have a pretty good idea of what gitolite does.
Once gitolite is set up, we will use use a variety of source repositories from Funtoo Linux to generate our own meta-repo and kits, which we can then test using Metro. This process is totally automated by the Funtoo Linux merge scripts.
This document assumes you have basic knowledge of ssh-keygen
and how to generate public/private SSH key pairs. If you don't know how to to this, see Generating SSH Keys for quick steps or OpenSSH Key Management, Part 1 for a more detailed introduction. For this article, you'll probably want to generate a private keys without a passphrase, which is more convenient but a much greater security risk if the private key gets compromised, or one with a passphrase but using keychain to manage ssh-agent for you.
Gitolite
Installation
To set up gitolite on your LAN, first choose a system that will be used to house your meta-repo and kits git repositories. You can do this on the same system you will be using for testing (and even development), or you can set it up on a dedicated system. It's actually fine to set this up anywhere on the Internet, as git will use ssh to access this repository, but for the purposes of this article, we're assuming you're setting it up somewhere on your LAN. We will refer to this system as repohost.
On this system, perform the following steps as root:
root # useradd -m repos
The repos
user will be a dedicated user account on the system that will have gitolite enabled and will house our git repositories. Now, we are going to su
to this new user on repohost and perform gitolite configuration:
root # su repos user $ git clone https://github.com/sitaramc/gitolite user $ install -d ~/bin
Now, as the repos
user, add the following to the end of your ~/.bashrc
file:
export PATH=$HOME/bin:$PATH
What we're doing is setting up a bin
directory where the gitolite
command will be installed, which will be in your path, so that you can use it more easily. With this done, perform the following steps:
user $ source ~/.bashrc user $ gitolite/install -ln
Now, your repos
account is almost ready to be used for hosting repositories. The way gitolite works is that it is going to basically take over ssh access to the account, so that when you connect via ssh with git, it will perform its own authentication. For this to work, you will need to enable your own "master key" to access gitolite. To do this, you'll want to decide from which account you'll want to administer gitolite itself. I prefer to use my "drobbins" account on my development workstation, so I will copy my ssh public key from ~/.ssh/id_rsa.pub
to /var/tmp/drobbins.pub
on the gitolite system, and then perform the following steps to "prime" gitolite with this admin public key:
user $ gitolite setup -pk /var/tmp/drobbins.pub
Gitolite will now be initialized to recognize the drobbins
remote account as an administrator. This is important because we will be performing the rest of gitolite setup over ssh, using this account.
OK, gitolite is installed on repohost! From this point forward, we will be using the drobbins
(or equivalent) account on your development workstation to configure gitolite remotely.
gitolite-admin Clone
Now that gitolite is ready, we can do everything else remotely. I am going to use the drobbins
account on my development workstation, and you will use whatever account is associated with the public key you loaded into gitolite. I like storing my development repos in /var/src
, so I'll go ahead and clone the gitolite-admin repo to that location so it can live along all my other git repos. Feel free to put this git repo wherever you like to store git repos that you develop on:
user $ cd /var/src user $ git clone repos@repohost:gitolite-admin user $ cd gitolite-admin
We are now ready to configure gitolite. We'll do this by modifying conf/gitolite.conf
in the git repo and adding new ssh public keys to keydir/
as needed. You will see that the initial public key you used to "prime" gitolite already exists in keydir/
. Once we change the configuration, and potentially add new public ssh keys that we want to grant access to gitolite-managed repositories, we'll perform a git commit
and git push
, and if gitolite doesn't complain about our changes, they'll take effect immediately. We'll go through our initial configuration steps below.
gitolite Configuration
To properly configure gitolite, we are going to need a public key that is associated with the root user of the workstation that is going to be generating meta-repo and kits. The system that I am using to do this is my personal development workstation, named ryzen
, so this key is going to be called ryzen-root.pub
.
You will want to copy this key into the keydir
directory in the gitolite-admin
repo, and add it to git:
user $ scp remote@blah:ryzen-root.pub keydir/ user $ git add keydir/ryzen-root.pub
Now, we're ready to edit conf/gitolite.conf
so that it looks like this:
gitolite.conf
# group = list of .pub keys in keydir:
@drobbins = drobbins
@repomgr = rzyen-root
# To enable read-only access to your meta-repo and kits, use this along with
# commented-out line under wildrepo. You will need to add box1-root.pub and
# box2-root.pub to keydir/ as well. This is good for boxes that will be testing
# your meta-repo and kits only but should not be able to modify them.
#@reporead = box1-root box2-root
# repositories:
# SPECIAL ADMIN REPO BELOW -- modify with care!
repo gitolite-admin
RW+ = @drobbins
# AUTO-CREATED (wild) REPOS: gitolite will auto-create repos under wildrepo/ for us
# upon initial clone of any path within, if the repo doesn't already exist.
repo wildrepo/..*
C = @repomgr
RW+ = @repomgr @drobbins
# NOTE: to enable read-only access for certain boxes, uncomment this line:
# R = @reporead
merge-scripts
Overview
Funtoo Linux uses "merge scripts" to create its kits. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.
The Code
You can find the code that does this on GitHub, housed at https://github.com/funtoo/merge-scripts. The script that does all the heavy-lifting is called merge-all-kits.py
. Let's clone it from git, on the machine that will be generating new kits and meta-repo. In our example, this will be as the root user on ryzen:
root # cd /root root # git clone https://github.com/funtoo/merge-scripts
While it is possible to use your own custom merge script repository, we recommend starting by using our official merge-scripts repository on GitHub, and progress to using your own fork of merge-scripts only when you need to.
Configuration
Now that merge-scripts is cloned, we will need to create a /root/.merge
configuration file. Use the following file as a starting point:
/root/.merge
[sources]
flora = https://github.com/funtoo/flora
kit-fixups = https://github.com/mygithub_user/kit-fixups
gentoo-staging = https://github.com/funtoo/gentoo-staging
[destinations]
base_url = repos@repohost:wildrepo/staging
[branches]
flora = master
kit-fixups = master
meta-repo = master
[work]
merge-scripts = /root/merge-scripts
source = /var/git/source-trees
destination = /var/git/dest-trees
Sources Section
Let's walk through this configuration file. The [sources]
section defines locations of repositories that the merge scripts will use as sources for creating kits and meta-repo. In the above sample config, we are using the official Flora repository from Funtoo, and the official gentoo-staging repository used by Funtoo, but we are using our own fork of https://github.com/funtoo/kit-fixups, which will allow us to add new ebuilds that will appear in kits, such as bug fixes to existing ebuilds in kits, as well as security fixes. For a core Funtoo Linux developer, this is a good way to start. If you are more interested in contributing third-party ebuilds, then you may instead choose to create your own fork of https://github.com/funtoo/flora, and use our standard kit-fixups repository. Or, you could choose to create forks of both. The recommended best practice is to use our upstream repos when possible, and fork only those repos you want to customize. This way, you'll ensure that you have the most up-to-date versions of ebuilds in those unforked repos.
Branches Section
The [branches]
section is used to define the default branches that are used by the merge-scripts. In general, sticking with master
is fine, but if you need the flexibility, you can point the merge scripts to a particular feature branch to use instead, for example.
Work Section
The [work]
section is used to define paths where the merge-scripts will do their work. merge-scripts
should point to the path where you cloned your merge-scripts repository, as merge-all-kits.py
will use this location to look for package-set definitions. These are files that define which catpkgs go in which kits. The source
and destination
settings above are good defaults, and define where the merge-scripts will clone source repositories and destination (written to) repositories, such as kits and meta-repo. These are kept in two separate hierarchies so they don't get mixed up.
Generating New Kits
With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be stored on repohost. Here are the steps you'd perform:
root # cd /root root # python3 merge-all-kits.py push
merge-all-kits.py
will proceed to create new kits and meta-repo, and will push them up to repos@repohost:wildrepo/staging/meta-repo, repos@repohost:wildrepo/staging/core-kit, etc. This process can take quite a while but has been optimized to run quickly on multi-core systems.
Using New Kits
Now that the new meta-repo and kits are created, here's how you'll use them on an existing Funtoo system, instead of your official Funtoo meta-repo and kits. First, we'll want to modify /etc/ego.conf
as follows:
/etc/ego.conf
[global]
sync_user = root
sync_base_url = repos@repohost:wildrepo/staging/{repo}
# You can have whatever [kits] section you want, below...
You will want to make sure that whatever system is connecting to repohost is permitted by gitolite, and has its /root/.ssh/id_rsa.pub file
stored and committed to the gitolite-admin repository's keydir/
and is referenced in conf/gitolite.conf
. Otherwise, gitolite will not allow this system to connect. Of course, if you are configuring this on the system that's running merge-all-kits.py
, it already has RW permissions to these repositories. Otherwise, you will want to make sure that gitolite has the system's keys as part of its @reporead
group.
We change the sync_user
to root
to force ego sync
to use /root/.ssh/id_rsa
for authentication. By default, ego sync
will attempt to use the portage
user which does not have a private key installed, and thus will not be able to authenticate with gitolite.
Now, let's move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run ego sync
:
root # cd /var/git root # mv meta-repo meta-repo.official root # ego sync
Ego will now sync your custom repository. If you type emerge -auDN @world
, ego will now be using your custom kits, rather than the official Funtoo ones :)