Note

The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.

Difference between revisions of "Funtoo:Metatools/Advanced Usage"

From Funtoo
Jump to navigation Jump to search
 
(119 intermediate revisions by 6 users not shown)
Line 1: Line 1:
This page documents how to set up a local development environment that will allow full local testing of any of your customizations, bug fixes or improvements to Funtoo Linux. This includes generating your own meta-repo and kits with your custom changes, as well as getting metro set up for unit tests. We recommend that all Funtoo Linux developers who will be submitting ebuild improvements to use set up this environment for local testing of changes. This setup also allows more involved changes to be tested and validated locally, so that your contributions to Funtoo are as production-ready as possible.
{{Subpages|Gitolite Setup}}
 
{{Important|These pages are out of date. Please see the latest [[Funtoo:Metatools]] page for usage information.}}


== Overview ==
== Overview ==


For our local development setup, we will be using [http://gitolite.com gitolite]. Gitolite will make things quite a bit easier by managing git repositories for us. ''Think of gitolite as your own private GitHub that has no Web user interface'' (we modify its settings by pushing to its special {{c|gitolite-admin}} repo) and you'll have a pretty good idea of what gitolite does. We will be using the following systems in these examples:
Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.
 
* {{c|repohost}} - this system will be running gitolite under the {{c|repos}} user account and will house git repositories for meta-repo and kits so that they are stored at a handy central location.
* {{c|rzyen}} - in these examples, this will be the primary development workstation, which will be used for editing cloned git code as well as generating custom kits. Once generated, the custom meta-repo and kits are pushed up to {{c|repohost}}.
 
{{Note|When you follow this guide, it is certainly possible to have {{c|repohost}} and {{c|ryzen}} be the same computer.}}
 
{{Important|This document assumes you have basic knowledge of {{c|ssh-keygen}} and how to generate public/private SSH key pairs. If you don't know how to to this, see [[Funtoo Containers#Generating SSH Keys|Generating SSH Keys]] for quick steps or [[OpenSSH Key Management, Part 1]] for a more detailed introduction. For this article, you'll probably want to generate a private keys without a passphrase, which is more convenient but a much greater security risk if the private key gets compromised, or one with a passphrase but using [[keychain]] to manage ssh-agent for you.}}
 
== Gitolite  ==
 
=== Installation ===
 
To set up gitolite on your LAN, first choose a system that will be used to house your meta-repo and kits git repositories. You can do this on the same system you will be using for testing (and even development), or you can set it up on a dedicated system. It's actually fine to set this up anywhere on the Internet, as git will use ssh to access this repository, but for the purposes of this article, we're assuming you're setting it up somewhere on your LAN.  We will refer to this system as '''repohost'''.
 
On this system, perform the following steps as root:
 
{{console|body=
# ##i##useradd -m repos
}}
 
The {{c|repos}} user will be a dedicated user account on the system that will have gitolite enabled and will house our git repositories. Now, we are going to {{c|su}} to this new user on '''repohost''' and perform gitolite configuration:
 
{{console|body=
# ##i##su repos
$ ##i##git clone https://github.com/sitaramc/gitolite
$ ##i##install -d ~/bin
}}
 
Now, as the {{c|repos}} user, add the following to the end of your {{c|~/.bashrc}} file:
 
{{file|name=~/.bashrc|body=
export PATH=$HOME/bin:$PATH
}}
 
What we're doing is setting up a {{c|bin}} directory where the {{c|gitolite}} command will be installed, which will be in your path, so that you can use it more easily. With this done, perform the following steps:


{{console|body=
Before starting, please read and understand the following pages:
$ ##i##source ~/.bashrc
* [[Kit-fixups]]
$ ##i##gitolite/install -ln
* [[Kit-fixups/FAQ]]
}}


Now, your {{c|repos}} account is almost ready to be used for hosting repositories. The way gitolite works is that it is going to basically take over ssh access to the account, so that when you connect via ssh with git, it will perform its own authentication. For this to work, you will need to enable your own "master key" to access gitolite. To do this, you'll want to decide from which account you'll want to administer gitolite itself. I prefer to use my "drobbins" account on my development workstation '''ryzen''', so I will copy my ssh public key from {{c|~/.ssh/id_rsa.pub}} to {{c|/var/tmp/ryzen-drobbins.pub}} on the gitolite system, and then perform the following steps to "prime" gitolite with this admin public key:
== Different Approaches ==


{{console|body=
{{c|merge-all-kits}} can be used in a variety of ways. If you are an individual developer, you may want to use {{c|merge-all-kits}} in ''developer mode'', which is its default mode. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.
$ ##i##gitolite setup -pk /var/tmp/ryzen-drobbins.pub
}}


Gitolite will now be initialized to recognize the {{c|drobbins}} remote account as an administrator, which will allow this remote account to clone from {{c|repos@repohost:gitolite-admin}} and push any changes to this special git repository which contains the master configuration for gitolite. This is important because we will be performing the rest of gitolite setup over ssh, using this account.
In developer mode, a {{c|meta-repo}} and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history.


{{Note|OK, gitolite is installed on '''repohost'''! From this point forward, we will be using the {{c|drobbins}} (or equivalent) account on your development workstation to configure gitolite remotely.}}
Another way to use {{c|merge-all-kits}} is with {{c|gitolite}} as a remote, which is documented under the [[#Gitolite Setup|Gitolite Setup]] sub-page. Using this configuration, {{c|merge-all-kits}} will write all of its destination git repositories to gitolite. You can then configure other systems to {{c|ego sync}} from gitolite as a source. This can be good in a distributed development environment. This method can also be used to create a meta-repo and kits that ''diverge'' from Funtoo's kit but still connect to its history.


=== gitolite-admin Clone ===
Finally, the {{c|merge-all-kits}} script can be run in a production mode, which is very similar to using it with gitolite except that you may be pushing up your destination meta-repo and kits to a public location like GitHub or GitLab.


Now that gitolite is ready on '''repohost''', we can do everything else remotely. I am going to use the {{c|drobbins}} account on my development workstation '''ryzen''', and you will use whatever account is associated with the public key you loaded into gitolite. I like storing my development repos in {{c|/var/src}} on '''ryzen''', so I'll go ahead and clone the gitolite-admin repo to that location so it can live along all my other git repos. Feel free to put this git repo wherever you like to store git repos that you develop on:
When first starting out with {{c|merge-all-kits}}, you should definitely start with developer mode and see if that meets your needs first.


{{console|body=
== Getting The Code ==
$ ##i##cd /var/src
$ ##i##git clone repos@repohost:gitolite-admin
$ ##i##cd gitolite-admin
}}
 
We are now ready to configure gitolite. We'll do this by modifying {{c|conf/gitolite.conf}} in the git repo and adding new ssh public keys to {{c|keydir/}} as needed. You will see that the initial public key you used to "prime" gitolite already exists in {{c|keydir/}}. Once we change the configuration, and potentially add new public ssh keys that we want to grant access to gitolite-managed repositories, we'll perform a {{c|git commit}} and {{c|git push}}, and if gitolite doesn't complain about our changes, they'll take effect immediately. We'll go through our initial configuration steps below.
 
=== gitolite Configuration ===
 
Since I will be generating meta-repo and kits on '''ryzen''', this system will need to have permissions to create repositories in gitolite. I would typically do this on '''ryzen''' as follows. First, as root:
 
{{console|body=
# ##i##cp /root/.ssh/id_rsa.pub /var/tmp
}}


Then, as my regular {{c|drobbins}} user that I typically use to perform development:
You can find the code that does this on code.funtoo.org, housed at [https://code.funtoo.org/bitbucket/projects/CORE/repos/merge-scripts/browse code.funtoo.org]. The script that does all the heavy-lifting is called {{c|merge-all-kits}}. Let's clone it from git, on the machine that will be generating new kits and meta-repo:


{{console|body=
{{console|body=
$ ##i##cd /var/src/gitolite-admin
$ ##i##git clone https://code.funtoo.org/bitbucket/scm/core/merge-scripts.git
$ ##i##cp /var/tmp/id_rsa.pub keydir/ryzen-root.pub
$ ##i##git add keydir/*
}}
}}


{{Note|It's important to change the filename of the public keys you are adding to the {{c|gitolite-admin}} repository. I typically use the format {{c|host-user.pub}}.}}
You will also want to install the following dependencies so that the code can run:
 
Now, we're ready to edit {{c|conf/gitolite.conf}} so that it looks like this:
 
{{file|name=gitolite.conf|body=
# Group definitions below, starting with @. This makes it easy to associate multiple ssh keys with a particular person.
 
@drobbins = ryzen-drobbins
@repomgr = rzyen-root
 
# To enable read-only access to your meta-repo and kits, use this along with
# commented-out line under wildrepo. You will need to add box1-root.pub and
# box2-root.pub to keydir/ as well. This is good for boxes that will be testing
# your meta-repo and kits only but should not be able to modify them.
 
#@reporead = box1-root box2-root
 
# repositories:
 
# SPECIAL ADMIN REPO BELOW -- modify with care! I've switched over to using the @drobbins group instead of
# referencing the individual ryzen-drobbins key directly.
 
repo gitolite-admin
    RW+    =  @drobbins
 
# AUTO-CREATED (wild) REPOS: gitolite will auto-create repos under wildrepo/ for us
# upon initial clone of any path within, if the repo doesn't already exist.
 
repo wildrepo/..*
    C      =  @repomgr
    RW+    =  @repomgr @drobbins
# NOTE: to enable read-only access for certain boxes, uncomment this line:
#  R      =  @reporead
}}
 
== merge-scripts ==
 
=== Overview ===
 
Funtoo Linux uses "merge scripts" to create its kits. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.
 
=== The Code ===
 
You can find the code that does this on GitHub, housed at https://github.com/funtoo/merge-scripts. The script that does all the heavy-lifting is called {{c|merge-all-kits.py}}. Let's clone it from git, on the machine that will be generating new kits and meta-repo. In our example, this will be as the root user on ryzen:


{{console|body=
{{console|body=
# ##i##cd /root
# ##i##emerge jinja lxml
# ##i##git clone https://github.com/funtoo/merge-scripts
}}
}}


{{Note|While it is possible to use your own custom merge script repository, we recommend starting by using our official merge-scripts repository on GitHub, and progress to using your own fork of merge-scripts only when you need to.}}
== Configuration and Paths ==


=== Configuration ===
In general, you should run {{c|merge-all-kits}} as a regular user, and this user needs to be a member of the {{c|portage}} group.


Now that merge-scripts is cloned, we will need to create a {{c|/root/.merge}} configuration file. Use the following file as a starting point:
In the user's home directory, create a {{f|~/.merge}} directory with the following contents:


{{file|name=/root/.merge|body=
{{file|name=/root/.merge|body=
[sources]
[sources]


flora = https://github.com/funtoo/flora
flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
kit-fixups = https://github.com/mygithub_user/kit-fixups
kit-fixups = https://code.funtoo.org/bitbucket/scm/co/flora.git
gentoo-staging = https://github.com/funtoo/gentoo-staging
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git
 
[destinations]
 
base_url = repos@repohost:wildrepo/staging


[branches]
[branches]
Line 155: Line 55:
kit-fixups = master
kit-fixups = master
meta-repo = master
meta-repo = master
[work]
source = /var/git/source-trees
destination = /var/git/dest-trees
}}
}}


==== Sources Section ====
By default, {{c|merge-all-kits}} will create a {{c|~/repo_tmp}} directory where it will store all of its work. Inside {{f|~/repo_tmp/source-trees}}, source git trees that will be used to generate meta-repo will be cloned. Inside {{f|~/repo_tmp/dest-trees}}, git repositories will be created to write results into. The meta-repo will be stored at {{c|/~repo_temp/dest-trees/meta-repo}}, and in developer mode, the kits will be stored in meta-repo, at {{c|~/repo_tmp/dest-trees/meta-repo/kits}}, so that meta-repo is configured 'ready to use' at it would appear at {{f|/var/git/meta-repo}}. When 'production' mode is specified via the {{c|--prod}} switch, the meta-repo will not have kits stacked inside it but instead the kits will be written directly into {{c|~/repo_tmp/dest-trees}}.
 
Let's walk through this configuration file. The {{c|[sources]}} section defines locations of repositories that the merge scripts will use as sources for creating kits and meta-repo. In the above sample config, we are using the official [[Flora]] repository from Funtoo, and the official gentoo-staging repository used by Funtoo, but we are using our own fork of https://github.com/funtoo/kit-fixups, which will allow us to add new ebuilds that will appear in kits, such as bug fixes to existing ebuilds in kits, as well as security fixes. For a core Funtoo Linux developer, this is a good way to start. If you are more interested in contributing third-party ebuilds, then you may instead choose to create your own fork of https://github.com/funtoo/flora, and use our standard kit-fixups repository. Or, you could choose to create forks of both. The recommended best practice is to use our upstream repos when possible, and fork only those repos you want to customize. This way, you'll ensure that you have the most up-to-date versions of ebuilds in those unforked repos.


==== Branches Section ====
A metadata cache will also be created and stored at {{c|~/repo_tmp/metadata-cache}}, and your user must be a member of the {{c|portage}} group for {{c|merge-all-scripts}} to be able to properly generate the metadata cache.


The {{c|[branches]}} section is used to define the default branches that are used by the merge-scripts. In general, sticking with {{c|master}} is fine, but if you need the flexibility, you can point the merge scripts to a particular feature branch to use instead, for example.
== Running Merge-All-Kits ==


==== Work Section ====
To run {{c|merge-all-kits}}, use it as follows:
 
The {{c|[work]}} section is used to define paths where the merge-scripts will do their work. The {{c|source}} and {{c|destination}} settings above are good defaults, and define where the merge-scripts will clone source repositories and destination (written to) repositories, such as kits and meta-repo. These are kept in two separate hierarchies so they don't get mixed up.
 
== Generating New Kits ==
 
With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be stored on '''repohost'''. Here are the steps you'd perform:


{{console|body=
{{console|body=
# ##i##cd /root/merge-scripts
$ ##i##path/to/merge-scripts/bin/merge-all-kits 1.4-release
# ##i##bin/merge-all-kits push
}}
}}


{{c|merge-all-kits}} will proceed to create new kits and meta-repo, and will push them up to repos@repohost:wildrepo/staging/meta-repo, repos@repohost:wildrepo/staging/core-kit, etc. This process can take quite a while but has been optimized to run quickly on multi-core systems.
The script will then run, and it will git clone flora, kit-fixups and gentoo-staging from the URIs in the {{c|~/.merge}} file, and assemble them into a complete meta-repo. The meta-repo will be located at {{f|$HOME/repo_tmp/dest-trees/meta-repo}}.
 
== Using New Kits ==


Now that the new meta-repo and kits are created, here's how you'll use them on an existing Funtoo system, instead of your official Funtoo meta-repo and kits. First, we'll want to modify {{c|/etc/ego.conf}} as follows:
To use the {{c|meta-repo}} for testing, you can add the following lines to {{f|/etc/ego.conf}}:


{{file|name=/etc/ego.conf|body=
{{file|name=/etc/ego.conf|body=
[global]
[global]
meta_repo_path = /home/user/repo_tmp/dest-trees/meta-repo
}}


sync_user = root
Then, using ego 2.8.0 or later, run the following command:
sync_base_url = repos@repohost:wildrepo/staging/{repo}


# Yes, you are supposed to have a literal "{repo}", above. Ego recognizes this special pattern.
{{console|body=
 
# ##i##ego sync --in-place
# You can have whatever [kits] section you want, below...
}}
}}


You will want to make sure that whatever system is connecting to '''repohost''' is permitted by gitolite, and has its {{c|/root/.ssh/id_rsa.pub file}} stored and committed to the gitolite-admin repository's {{c|keydir/}} and is referenced in {{c|conf/gitolite.conf}}. Otherwise, gitolite will not allow this system to connect. Of course, if you are configuring this on the system that's running {{c|merge-all-kits.py}}, it already has RW permissions to these repositories. Otherwise, you will want to make sure that gitolite has the system's keys as part of its {{c|@reporead}} group.
This will ensure that the proper kit branches in your meta-repo are checked out and that all the repos.conf and profiles files in {{c|/etc/portage}} are configured correctly for your repository to be used. Note that no actual "{{c|git pull}}" will be performed -- this is intentional and a special feature of the {{c|--in-place}} option -- because your local git repositories have no remotes to pull from and contain all the information they need to be used.
 
{{Note|We change the {{c|sync_user}} to {{c|root}} to force {{c|ego sync}} to use {{c|/root/.ssh/id_rsa}} for authentication. By default, {{c|ego sync}} will attempt to use the {{c|portage}} user which does not have a private key installed, and thus will not be able to authenticate with gitolite. Rather than mess with the {{c|portage}} user and give it a proper home directory and ssh key pair, it's easier just to not drop perms to {{c|portage}} in the first place.}}


Now, let's move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run {{c|ego sync}}:
You're now ready to use your meta-repo directly:


{{console|body=
{{console|body=
# ##i##cd /var/git
# ##i##emerge foo
# ##i##mv meta-repo meta-repo.official
# ##i##ego sync
}}
}}


Ego will now sync your custom repository. If you type {{c|emerge -auDN @world}}, ego will now be using your custom kits, rather than the official Funtoo ones. This means that you can perform a variety of things you couldn't before. You can now add your own custom ebuilds to your fork of {{c|kit-fixups}}, and {{c|merge-all-kits.py}} will automatically incorporate these changes into your own custom kits. This will allow you to locally test any changes before submitting them as pull requests to Funtoo. You will also be able to maintain your own meta-repo and kits with your own local modifications, and have your systems use these meta-repo/kits instead of the official Funtoo ones.
[[Category:Official Documentation]]
[[Category:Development]]

Latest revision as of 21:47, June 8, 2022

Subpages:
Gitolite Setup
   Important

These pages are out of date. Please see the latest Funtoo:Metatools page for usage information.

Overview

Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.

Before starting, please read and understand the following pages:

Different Approaches

merge-all-kits can be used in a variety of ways. If you are an individual developer, you may want to use merge-all-kits in developer mode, which is its default mode. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.

In developer mode, a meta-repo and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history.

Another way to use merge-all-kits is with gitolite as a remote, which is documented under the Gitolite Setup sub-page. Using this configuration, merge-all-kits will write all of its destination git repositories to gitolite. You can then configure other systems to ego sync from gitolite as a source. This can be good in a distributed development environment. This method can also be used to create a meta-repo and kits that diverge from Funtoo's kit but still connect to its history.

Finally, the merge-all-kits script can be run in a production mode, which is very similar to using it with gitolite except that you may be pushing up your destination meta-repo and kits to a public location like GitHub or GitLab.

When first starting out with merge-all-kits, you should definitely start with developer mode and see if that meets your needs first.

Getting The Code

You can find the code that does this on code.funtoo.org, housed at code.funtoo.org. The script that does all the heavy-lifting is called merge-all-kits. Let's clone it from git, on the machine that will be generating new kits and meta-repo: 

user $ git clone https://code.funtoo.org/bitbucket/scm/core/merge-scripts.git

You will also want to install the following dependencies so that the code can run: 

root # emerge jinja lxml

Configuration and Paths

In general, you should run merge-all-kits as a regular user, and this user needs to be a member of the portage group.

In the user's home directory, create a ~/.merge directory with the following contents:

   /root/.merge 
[sources]

flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
kit-fixups = https://code.funtoo.org/bitbucket/scm/co/flora.git
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git

[branches]

flora = master
kit-fixups = master
meta-repo = master

By default, merge-all-kits will create a ~/repo_tmp directory where it will store all of its work. Inside ~/repo_tmp/source-trees, source git trees that will be used to generate meta-repo will be cloned. Inside ~/repo_tmp/dest-trees, git repositories will be created to write results into. The meta-repo will be stored at /~repo_temp/dest-trees/meta-repo, and in developer mode, the kits will be stored in meta-repo, at ~/repo_tmp/dest-trees/meta-repo/kits, so that meta-repo is configured 'ready to use' at it would appear at /var/git/meta-repo. When 'production' mode is specified via the --prod switch, the meta-repo will not have kits stacked inside it but instead the kits will be written directly into ~/repo_tmp/dest-trees.

A metadata cache will also be created and stored at ~/repo_tmp/metadata-cache, and your user must be a member of the portage group for merge-all-scripts to be able to properly generate the metadata cache.

Running Merge-All-Kits

To run merge-all-kits, use it as follows: 

user $ path/to/merge-scripts/bin/merge-all-kits 1.4-release

The script will then run, and it will git clone flora, kit-fixups and gentoo-staging from the URIs in the ~/.merge file, and assemble them into a complete meta-repo. The meta-repo will be located at $HOME/repo_tmp/dest-trees/meta-repo.

To use the meta-repo for testing, you can add the following lines to /etc/ego.conf:

   /etc/ego.conf 
[global]
meta_repo_path = /home/user/repo_tmp/dest-trees/meta-repo

Then, using ego 2.8.0 or later, run the following command: 

root # ego sync --in-place

This will ensure that the proper kit branches in your meta-repo are checked out and that all the repos.conf and profiles files in /etc/portage are configured correctly for your repository to be used. Note that no actual "git pull" will be performed -- this is intentional and a special feature of the --in-place option -- because your local git repositories have no remotes to pull from and contain all the information they need to be used.

You're now ready to use your meta-repo directly: 

root # emerge foo
Retrieved from "https://www.funtoo.org/index.php?title=Funtoo:Metatools/Advanced_Usage&oldid=40517"