Funtoo:Metatools/Advanced Usage

Subpages:

Gitolite Setup

Kit-Fixups Kit Foundations

Package Sets and Move Maps

Overview

Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.

Required for merge scripts is dev-python/lxml.

Different Approaches

merge-all-kits can be used in a variety of ways. If you are an individual developer, you may want to use merge-all-kits in developer mode, which is its default configuration. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.

In developer mode, a meta-repo and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk locally, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history. When

Another way to use merge-all-kits is with gitolite as a remote, which is documented under the Gitolite Setup sub-page. Using this configuration, merge-all-kits will write all of its destination git repositories to gitolite. You can then configure other systems to ego sync from gitolite as a source. This can be good in a distributed development environment. This method can also be used to create a meta-repo and kits that diverge from Funtoo's kit but still connect to its history.

Finally, the merge-all-kits script can be run in a production mode, which is very similar to using it with gitolite except that you may be pushing up your destination meta-repo and kits to a public location like GitHub or GitLab.

When first starting out with merge-all-kits, you should definitely start with developer mode and see if that meets your needs first.

Getting The Code

You can find the code that does this on code.funtoo.org, housed at https://code.funtoo.org/bitbucket/projects/CORE/repos/merge-scripts/browse. The script that does all the heavy-lifting is called merge-all-kits. Let's clone it from git, on the machine that will be generating new kits and meta-repo:

user $ git clone https://code.funtoo.org/bitbucket/scm/core/merge-scripts.git

You will also want to install the following dependencies so that the code can run:

root # emerge jinja lxml

Configuration

In general, you should run merge-all-kits as a regular user, and this user needs to be a member of the portage group.

In the user's home directory, create a ~/.merge directory with the following contents:

   /root/.merge

[sources]

flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
kit-fixups = https://code.funtoo.org/bitbucket/scm/co/flora.git
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git

[branches]

flora = master
kit-fixups = master
meta-repo = master

Running Merge-All-Kits

To run merge-all-kits, use it as follows:

user $ path/to/merge-scripts/bin/merge-all-kits 1.4-release

The script will then run, and it will git clone flora, kit-fixups and gentoo-staging from the URIs in the ~/.merge file, and assemble them into a complete meta-repo. The meta-repo will be located at $HOME/repo_tmp/dest-trees/meta-repo.

To use the meta-repo for testing, you can add the following lines to /etc/ego.conf:

   /etc/ego.conf

[global]
meta_repo_path = /home/user/repo_tmp/dest-trees/meta-repo

Then, using ego 2.8.0 or later, run the following command:

root # ego sync --in-place

This will ensure that the proper kit branches in your meta-repo are checked out and that all the repos.conf and profiles files in /etc/portage are configured correctly for your repository to be used. Note that no actual "git pull" will be performed -- this is intentional and a special feature of the --in-place option -- because your local git repositories have no remotes to pull from and contain all the information they need to be used.

You're now ready to use your meta-repo directly:

root # emerge foo

Package Sets

Package sets define which catpkgs go in which kits. The package set files are located at kit-fixups/package-sets. They can either consist of a single file named (kit)-packages or a directory with the same name. In the case of a directory, all of the files inside the directory are concatenated and used as a package set.

Package Set Format

Package sets are a text-based format that consist of one catpkg entry per line. Let's look at the various types of package sets entries:

Entry type	Example	Explanation
literal	sys-apps/portage	Specify a single catpkg by exact name.
literal with move	sys-apps/oldpkg -> sys-foo/newpkg	Specify a catpkg by exact name, but copy it over as a new name.
category wildcard	sys-apps/*	Specify all packages that appear in a particular category.
category wildcard with exceptions	sys-apps/* -sys-apps/foo -sys-apps/bar	Specify all packages that appear in a particular category, with some exceptions.
regex	sys-.*/foo.*	Specify all catpkgs that match a particular regex.
dependencies in category	@depsincat@:x11-base/xorg-x11:media-fonts	In this example, anything in the media-fonts category that has dependencies upon x11-base/xorg-x11.
maintainer	@maintainer@:dev-lang:ml@gentoo.org	In this example, all dev-lang packages that have a maintainer of ml@gentoo.org.
has eclass	@has_eclass@:kde5	In this example, all catpkgs that use an eclass of kde5.eclass.
category has eclass	@cat_has_eclass@:x11-apps:xorg-2	In this example, all catpkgs in the x11-apps category that use an eclass of xorg-2.eclass.

   Important

It is possible to blacklist certain catpkgs for inclusion in particular kit by creating a "skip" list at kit-fixups/package-sets/mykitnamehere-packages-skip. This file can contain individual catpkgs, one per line, that should not be included in the package-set, even if they match any rules above. Note that you must use literal catpkgs in the skip list -- no patterns or other special matches are supported.

Inline Package Moves

   Note

The "literal with move" option is a new feature, described below.

The "literal with move" package set syntax described above is one way to tell the merge scripts to copy a catpkg from a source location but give it a new catpkg name, and they are specified directly in the package-set files.

If the old package name is found, it will be copied over as the new name. If the old name is not found, but the new name is found, the new name will be copied over as the new name. So either the old name or the new name, if found, will be copied over.

Move-Maps

You can also specify package moves by creating a file called kit-fixups/move-maps/kitname containing the same "literal with move" syntax. Global move maps can be placed in kit-fixups/move-maps/global. Also note that kit-fixups/move-maps/nokit or kit-fixups/move-maps/global is the only way to perform funtoo package moves for nokit. I recommend using the "global" method since it will still automatically work if someone else moves your package into a kit.

This move-maps functionality works similarly to package-moves that appear within a kit package-set, except that these moves do not automatically add either specified catpkg to the kit. So there must be something in the package-set that matches the old package name. After the old package is matched, additional logic looks at the move-maps and see if the match is in a move-map that tells us to copy it over as the "new" name. The old package will be copied over as the new name.

If you need to just rename a single package in a kit, it's fine to use the inline method. For more capability, move-maps/ are your friend. They're more powerful because a move-map specified with move-maps/ will also apply to any special wildcard package-set matches via @regex@, etc, whereas the inline method is limited by design. Also, the out-of-band method allows you to perform package moves on catpkgs in nokit.

   Important

Package moves give you the ability to rename catpkgs as they appear in kits. We still need to add functionality to provide this data to Portage so that it can update any package database entries for packages installed under the old name. This part is not done yet, so this is considered a testing-only feature for the time being and should only be used on local kit-fixups overlays for testing, not in our official kit-fixups repo yet.

Package Sets -- Putting It All Together

Here are some important facts about package sets:

1. Package set matches are executed in a particular order, and this order is defined by the order of kits in kit_groups in foundations.py.
2. Once a catpkg is matched during processing of a kit, that catpkg is assigned to that kit, and cannot appear in another kit.
3. If a catpkg is included in a particular branch of a kit, then that catpkg will appear in all branches of that kit, assuming it is available.
4. Each kit and branch defined in kit_groups specifies a "source" -- an entry in kit_source_defs which in turn defines a stack of repositories/overlays and associated SHA1 commits to use as sources for catpkgs.

So, here's how the package set processing would begin. If we look at kit_groups in foundations.py, we see that core-kit 1.0-prime is listed first. So we will look for catpkg matches for core-kit 1.0-prime using the kit-fixups/package-sets/core-kit-packages package set directory. We will apply these match rules against the gentoo_prime_protected kit source definition, and we will look for matches in each repository in the order listed in the kit_source_defs entry. The first match we find will be used as the source catpkg. But remember that we have specific rules in place, defined in the overlays property, that effectively gives gentoo-staging priority for most of the catpkgs.

After this is done, we will then process core-kit 1.2-prime, since 1.1-prime is deprecated and will be skipped, and then continue to core-hw-kit, and continue to work down the kit_groups list. This process will build up a set of catpkgs that will appear in each kit.

Kit Fixups

And finally, we have saved a very key part of the kit generation process for last. The kit-fixups repository is so named because it contains fixups, which are forked Funtoo catpkgs that are used to override catpkgs that appear in the upstream overlays and repositories. They have a special structure. We will look at the structure of the core-kit fixup directory, although others will follow the same model:

CatPkg Path	Description
kit-fixups/core-kit/global/sys-apps/portage	Due to the global directory, this catpkg will always be used when a package set specifies a match for sys-apps/portage, for all branches of core-kit.
kit-fixups/core-kit/curated/sys-apps/portage	Due to the curated directory, this catpkg will always be used when a package set specifies a match for sys-apps/portage, for all branches of core-kit except a master branch.
kit-fixups/core-kit/1.2-prime/sys-apps/portage	Due to the 1.2-prime directory, this catpkg will always be used when a package set specifies a match for sys-apps/portage, for the 1.2-prime branch of core-kit only.

Remember that kit-fixups is designed so that a fixup will always override any upstream packages. This makes it easy to keep track of Funtoo-maintained core packages. And also note that the flora repository should be used for "bonus" packages while kit-fixups should focus more on forks of critical system packages and bug fixes for Funtoo. This way, we can keep contributed ebuilds separate from core operating system ebuilds and associated bug fixes for upstream issues.

Developer Q&A

This section contains various tasks that a developer may need to perform, and what steps should be taken to perform each of these steps.

I want to move a catpkg sys-apps/blah from core-kit to foobar-kit.

To do this, first we'll note that core-kit comes before foobar-kit in kit_groups. This means that core-kit's package set rules will run first. So we will want to make sure that sys-apps/blah does not match any rules in the core-kit package-set. This can be done by possibly removing a package-set rule, or using a wildcard with exclusion like sys-apps/* -sys-apps/blah. If this doesn't work, a file can be created called core-kit-packages-skip which contains exclusions, and sys-apps/blah can be added to a line in this file. Then, you will want to make sure that sys-apps/blah does match a package set rule for foobar-kit.

I want to move a catpkg sys-apps/blah from foobar-kit to core-kit.

To do this, first we'll note that foobar-kit comes after core-kit in kit-groups, so core-kit's package set rules will run first. We can thus simply add something that will match 'sys-apps/blah' to core-kit's package-set rules. Once sys-apps/blah is included in core-kit, it will not be available for inclusion in foobar-kit, even if it has an identical rule, or a rule like 'sys-apps/*'. However, note that it is good practice to clean up any rules in foobar-kit that you know are no longer matching any catpkgs.

   Note

The above two approaches can be used to move catpkgs between kits transparently to the end-user. In the next ego sync, the catpkg will atomically move from one kit to another and no re-emerging will be required, even if the user had emerged the package from the 'old' kit location.

I want to contribute a cool package to Funtoo.

To do this, you will want to open a pull request against flora. Flora is used for all 'bonus' community-contributed ebuilds.

I want to fix a bug in a particular ebuild.

To do this, first find out where the ebuild is coming from. A good way to do this is to type ls -d /var/git/meta-repo/kits/*/sys-apps/foobar, which will show you what kit it is in. Running emerge -s sys-apps/foobar will also display this information. For research purposes, it is often useful to find where the original catpkg was sourced from. You can consult https://ports.funtoo.org/packages.xml which contains a list of all catpkgs and their source repository. After doing some initial research and seeing what's wrong, you might have a fix for the ebuild. Generally, the best way to fix the ebuild is to fork kit-fixups and create an appropriate fixup for the ebuild if none exists, and simply improve our fixup if one exists already. Then you can create a code.funtoo.org pull request, or open a bug on bugs.funtoo.org, or both. Remember that fixup catpkgs will totally replace all upstream ebuilds, so you may need to include multiple versions of the ebuild, even ones that don't need a fix, if they are still needed for certain packages.

   Note

If you want to fix a bug in an ebuild and you find that the ebuild comes from flora, you will want to fork flora and submit a pull request against flora instead.

I want to make a particular branch of a kit the default kit.

To do this, you will modify kit_groups and set the kit you want to be default to have 'default' : True or 'stability' : KitStabilityRating.PRIME, or both. Only one kit branch can be set as default.

I don't want to generate a particular branch of a kit.

To prevent a branch of a kit from being generated, set its stability to KitStabilityRating.DEPRECATED in kit_groups.

I want to generate a new kit branch that uses much newer ebuilds from Gentoo or from an upstream repo.

First, define a new entry in kit_source_defs that contains the collection of overlays and repos you want to use as sources. Specify the SHA1 commits you want to use for each repo (or don't specify one to use master.) Then, you will want to add a new kit definition to kit_groups, in the "prime" section.

I want to include a package in Funtoo, but move it to a new name.

To do this, use the "literal with package move" format in the package set (see section on #Package Moves, above). You can also use files in kit_fixups/move-maps/kitname or kit_fixups/move-maps/global (info in a note below the Package Set syntax section.).

   Important

Be sure to stop by #funtoo-dev on irc.freenode.net if you need further assistance! We are here to help.