Funtoo:Metatools/Advanced Usage

This page documents how to set up a local development environment that will allow full local testing of any of your customizations, bug fixes or improvements to Funtoo Linux. This includes generating your own meta-repo and kits with your custom changes, as well as getting metro set up for unit tests. We recommend that all Funtoo Linux developers who will be submitting ebuild improvements to use set up this environment for local testing of changes. This setup also allows more involved changes to be tested and validated locally, so that your contributions to Funtoo are as production-ready as possible.

Overview

For our local development setup, we will be using gitolite. Gitolite will make things quite a bit easier by managing git repositories for us. Think of gitolite as your own private GitHub that has no Web user interface (we modify its settings by pushing to its special gitolite-admin repo) and you'll have a pretty good idea of what gitolite does. We will be using the following systems in these examples:

• repohost - this system will be running gitolite under the repos user account and will house git repositories for meta-repo and kits so that they are stored at a handy central location.
• ryzen - in these examples, this will be the primary development workstation, which will be used for editing cloned git code as well as generating custom kits. Once generated, the custom meta-repo and kits are pushed up to repohost.

   Note

When you follow this guide, it is certainly possible to have repohost and ryzen be the same computer.

   Important

This document assumes you have basic knowledge of ssh-keygen and how to generate public/private SSH key pairs. If you don't know how to to this, see Generating SSH Keys for quick steps or OpenSSH Key Management, Part 1 for a more detailed introduction. For this article, you'll probably want to generate a private keys without a passphrase, which is more convenient but a much greater security risk if the private key gets compromised, or one with a passphrase but using keychain to manage ssh-agent for you.

Gitolite

Installation

To set up gitolite on your LAN, first choose a system that will be used to house your meta-repo and kits git repositories. You can do this on the same system you will be using for testing (and even development), or you can set it up on a dedicated system. It's actually fine to set this up anywhere on the Internet, as git will use ssh to access this repository, but for the purposes of this article, we're assuming you're setting it up somewhere on your LAN. We will refer to this system as repohost.

On this system, perform the following steps as root:

root # useradd -m repos

The repos user will be a dedicated user account on the system that will have gitolite enabled and will house our git repositories. Now, we are going to su to this new user on repohost and perform gitolite configuration:

root # su repos
user $ git clone https://github.com/sitaramc/gitolite
user $ install -d ~/bin

Now, as the repos user, add the following to the end of your ~/.bashrc file:

   ~/.bashrc

export PATH=$HOME/bin:$PATH

What we're doing is setting up a bin directory where the gitolite command will be installed, which will be in your path, so that you can use it more easily. With this done, perform the following steps:

user $ source ~/.bashrc
user $ gitolite/install -ln

Now, your repos account is almost ready to be used for hosting repositories. The way gitolite works is that it is going to basically take over ssh access to the account, so that when you connect via ssh with git, it will perform its own authentication. For this to work, you will need to enable your own "master key" to access gitolite. To do this, you'll want to decide from which account you'll want to administer gitolite itself. I prefer to use my "drobbins" account on my development workstation ryzen, so I will copy my ssh public key from ~/.ssh/id_rsa.pub to /var/tmp/ryzen-drobbins.pub on the gitolite system, and then perform the following steps to "prime" gitolite with this admin public key:

user $ gitolite setup -pk /var/tmp/ryzen-drobbins.pub

Gitolite will now be initialized to recognize the drobbins remote account as an administrator, which will allow this remote account to clone from repos@repohost:gitolite-admin and push any changes to this special git repository which contains the master configuration for gitolite. This is important because we will be performing the rest of gitolite setup over ssh, using this account.

   Note

OK, gitolite is installed on repohost! From this point forward, we will be using the drobbins (or equivalent) account on your development workstation to configure gitolite remotely.

gitolite-admin Clone

Now that gitolite is ready on repohost, we can do everything else remotely. I am going to use the drobbins account on my development workstation ryzen, and you will use whatever account is associated with the public key you loaded into gitolite. I like storing my development repos in /var/src on ryzen, so I'll go ahead and clone the gitolite-admin repo to that location so it can live along all my other git repos. Feel free to put this git repo wherever you like to store git repos that you develop on:

user $ cd /var/src
user $ git clone repos@repohost:gitolite-admin
user $ cd gitolite-admin

We are now ready to configure gitolite. We'll do this by modifying conf/gitolite.conf in the git repo and adding new ssh public keys to keydir/ as needed. You will see that the initial public key you used to "prime" gitolite already exists in keydir/. Once we change the configuration, and potentially add new public ssh keys that we want to grant access to gitolite-managed repositories, we'll perform a git commit and git push, and if gitolite doesn't complain about our changes, they'll take effect immediately. We'll go through our initial configuration steps below.

gitolite Configuration

Since I will be generating meta-repo and kits on ryzen, this system will need to have permissions to create repositories in gitolite. I would typically do this on ryzen as follows. First, as root:

root # cp /root/.ssh/id_rsa.pub /var/tmp

Then, as my regular drobbins user that I typically use to perform development:

user $ cd /var/src/gitolite-admin
user $ cp /var/tmp/id_rsa.pub keydir/ryzen-root.pub
user $ git add keydir/*

   Note

It's important to change the filename of the public keys you are adding to the gitolite-admin repository. I typically use the format host-user.pub.

Now, we're ready to edit conf/gitolite.conf so that it looks like this:

   gitolite.conf

# Group definitions below, starting with @. This makes it easy to associate multiple ssh keys with a particular person.

@drobbins = ryzen-drobbins
@repomgr = ryzen-root

# To enable read-only access to your meta-repo and kits, use this along with
# commented-out line under wildrepo. You will need to add box1-root.pub and
# box2-root.pub to keydir/ as well. This is good for boxes that will be testing
# your meta-repo and kits only but should not be able to modify them.

#@reporead = box1-root box2-root

# repositories:

# SPECIAL ADMIN REPO BELOW -- modify with care! I've switched over to using the @drobbins group instead of 
# referencing the individual ryzen-drobbins key directly.

repo gitolite-admin
    RW+     =   @drobbins

# AUTO-CREATED (wild) REPOS: gitolite will auto-create repos under wildrepo/ for us
# upon initial clone of any path within, if the repo doesn't already exist.

repo wildrepo/..*
    C       =   @repomgr
    RW+     =   @repomgr @drobbins
# NOTE: to enable read-only access for certain boxes, uncomment this line:
#   R       =   @reporead

merge-scripts

Overview

Funtoo Linux uses "merge scripts" to create its kits. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.

Required for merge scripts is dev-python/lxml.

The Code

You can find the code that does this on GitHub, housed at https://github.com/funtoo/merge-scripts. The script that does all the heavy-lifting is called merge-all-kits.py. Let's clone it from git, on the machine that will be generating new kits and meta-repo. In our example, this will be as the root user on ryzen:

root # cd /root
root # git clone https://github.com/funtoo/merge-scripts

   Note

While it is possible to use your own custom merge script repository, we recommend starting by using our official merge-scripts repository on GitHub, and progress to using your own fork of merge-scripts only when you need to.

Configuration

Now that merge-scripts is cloned, we will need to create a /root/.merge configuration file. Use the following file as a starting point:

   /root/.merge

[sources]

flora = https://github.com/funtoo/flora
kit-fixups = https://github.com/mygithub_user/kit-fixups
gentoo-staging = https://github.com/funtoo/gentoo-staging

[destinations]

base_url = repos@repohost:wildrepo/staging

[branches]

flora = master
kit-fixups = master
meta-repo = master

[work]

source = /var/git/source-trees
destination = /var/git/dest-trees

Sources Section

Let's walk through this configuration file. The [sources] section defines locations of repositories that the merge scripts will use as sources for creating kits and meta-repo. In the above sample config, we are using the official Flora repository from Funtoo, and the official gentoo-staging repository used by Funtoo, but we are using our own fork of https://github.com/funtoo/kit-fixups, which will allow us to add new ebuilds that will appear in kits, such as bug fixes to existing ebuilds in kits, as well as security fixes. For a core Funtoo Linux developer, this is a good way to start. If you are more interested in contributing third-party ebuilds, then you may instead choose to create your own fork of https://github.com/funtoo/flora, and use our standard kit-fixups repository. Or, you could choose to create forks of both. The recommended best practice is to use our upstream repos when possible, and fork only those repos you want to customize. This way, you'll ensure that you have the most up-to-date versions of ebuilds in those unforked repos.

Branches Section

The [branches] section is used to define the default branches that are used by the merge-scripts. In general, sticking with master is fine, but if you need the flexibility, you can point the merge scripts to a particular feature branch to use instead, for example.

Work Section

The [work] section is used to define paths where the merge-scripts will do their work. The source and destination settings above are good defaults, and define where the merge-scripts will clone source repositories and destination (written to) repositories, such as kits and meta-repo. These are kept in two separate hierarchies so they don't get mixed up.

Generating New Kits

With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be stored on repohost. Here are the steps you'd perform:

root # cd /root/merge-scripts
root # bin/merge-all-kits push

Before starting the script for the first time you should configure your git user.name and user.email variables.

root # git config --global user.email "you@example.com"
root # git config --global user.name "Your Name"

merge-all-kits will proceed to create new kits and meta-repo, and will push them up to repos@repohost:wildrepo/staging/meta-repo, repos@repohost:wildrepo/staging/core-kit, etc. This process can take quite a while but has been optimized to run quickly on multi-core systems.

Using New Kits

Now that the new meta-repo and kits are created, here's how you'll use them on an existing Funtoo system, instead of your official Funtoo meta-repo and kits. First, we'll want to modify /etc/ego.conf as follows:

   /etc/ego.conf

[global]

sync_user = root
sync_base_url = repos@repohost:wildrepo/staging/{repo}

# Yes, you are supposed to have a literal "{repo}", above. Ego recognizes this special pattern.

# You can have whatever [kits] section you want, below...

You will want to make sure that whatever system is connecting to repohost is permitted by gitolite, and has its /root/.ssh/id_rsa.pub file stored and committed to the gitolite-admin repository's keydir/ and is referenced in conf/gitolite.conf. Otherwise, gitolite will not allow this system to connect. Of course, if you are configuring this on the system that's running merge-all-kits.py, it already has RW permissions to these repositories. Otherwise, you will want to make sure that gitolite has the system's keys as part of its @reporead group.

   Note

We change the sync_user to root to force ego sync to use /root/.ssh/id_rsa for authentication. By default, ego sync will attempt to use the portage user which does not have a private key installed, and thus will not be able to authenticate with gitolite. Rather than mess with the portage user and give it a proper home directory and ssh key pair, it's easier just to not drop perms to portage in the first place.

Now, let's move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run ego sync:

root # cd /var/git
root # mv meta-repo meta-repo.official
root # ego sync

Ego will now sync your custom repository. If you type emerge -auDN @world, ego will now be using your custom kits, rather than the official Funtoo ones. This means that you can perform a variety of things you couldn't before. You can now add your own custom ebuilds to your fork of kit-fixups, and merge-all-kits.py will automatically incorporate these changes into your own custom kits. This will allow you to locally test any changes before submitting them as pull requests to Funtoo. You will also be able to maintain your own meta-repo and kits with your own local modifications, and have your systems use these meta-repo/kits instead of the official Funtoo ones.

Changing Repo Definitions

Now that you have generated your first meta-repo, let's look at the key file that defines what overlays and repositories are used to create meta-repo, as well as what kit branches exist, and which ones are prime and which are not. You will want to turn your attention to the kit-fixups/modules/fixups/foundations.py file, which can be viewed here on GitHub]. Let's take a look at various sections of this file:

   foundations.py (python source code) - Top of foundations.py file

#!/usr/bin/python3

from enum import Enum


class KitStabilityRating(Enum):
	PRIME = 0  # Kit is enterprise-quality
	NEAR_PRIME = 1  # Kit is approaching enterprise-quality
	BETA = 2  # Kit is in beta
	ALPHA = 3  # Kit is in alpha
	DEV = 4  # Kit is newly created and in active development
	CURRENT = 10  # Kit follows Gentoo currrent
	DEPRECATED = 11  # Kit is deprecated/retired


def KitRatingString(kit_enum):
	if kit_enum is KitStabilityRating.PRIME:
		return "prime"
	elif kit_enum is KitStabilityRating.NEAR_PRIME:
		return "near-prime"
	elif kit_enum is KitStabilityRating.BETA:
		return "beta"
	elif kit_enum is KitStabilityRating.ALPHA:
		return "alpha"
	elif kit_enum is KitStabilityRating.DEV:
		return "dev"
	elif kit_enum is KitStabilityRating.CURRENT:
		return "current"
	elif kit_enum is KitStabilityRating.DEPRECATED:
		return "deprecated"

At the top of the file, we define an enumeration of the different levels of stability that can exist for a particular kit. Kits marked with a stability rating of KitStabilityRating.PRIME will be tagged in the meta-repo JSON as being a "prime" kit and suitable for production use.

As you scroll down further, you will see some code like this:

   foundations.py (python source code) - foundations.py kit_groups

class KitFoundation:

	kit_groups = {
		'prime': [
			{'name': 'core-kit', 'branch': '1.0-prime', 'source': 'gentoo_prime_protected', 'default': True},
			{'name': 'core-kit', 'branch': '1.1-prime', 'source': 'gentoo_prime_mk3_protected', 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'core-kit', 'branch': '1.2-prime', 'source': 'gentoo_prime_mk4_protected', 'stability': KitStabilityRating.BETA},
			{'name': 'core-hw-kit', 'branch': 'master', 'source': 'funtoo_current', 'default': True},
			{'name': 'security-kit', 'branch': '1.0-prime', 'source': 'gentoo_prime_protected', 'default': True},
			{'name': 'security-kit', 'branch': '1.1-prime', 'source': 'gentoo_prime_mk3_protected', 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'security-kit', 'branch': '1.2-prime', 'source': 'gentoo_prime_mk4_protected', 'stability': KitStabilityRating.BETA},
			{'name': 'xorg-kit', 'branch': '1.17-prime', 'source': 'funtoo_prime_xorg', 'default': False, 'stability': KitStabilityRating.PRIME},
			{'name': 'xorg-kit', 'branch': '1.19-prime', 'source': 'funtoo_mk2_prime', 'default': True, 'stability': KitStabilityRating.PRIME},  # MK2
			{'name': 'gnome-kit', 'branch': '3.20-prime', 'source': 'funtoo_prime_gnome', 'default': True},
			{'name': 'gnome-kit', 'branch': '3.26-prime', 'source': 'funtoo_mk4_prime', 'default': False, 'stability': KitStabilityRating.DEV},
			{'name': 'kde-kit', 'branch': '5.10-prime', 'source': 'funtoo_mk3_prime', 'default': False, 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'kde-kit', 'branch': '5.11-prime', 'source': 'funtoo_prime_kde', 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'kde-kit', 'branch': '5.12-prime', 'source': 'funtoo_prime_kde_late', 'default': True, 'stability': KitStabilityRating.PRIME},
			{'name': 'media-kit', 'branch': '1.0-prime', 'source': 'funtoo_prime_media', 'default': False, 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'media-kit', 'branch': '1.1-prime', 'source': 'funtoo_mk3_prime', 'default': True, 'stability': KitStabilityRating.PRIME},  # MK3
			{'name': 'media-kit', 'branch': '1.2-prime', 'source': 'funtoo_mk4_prime', 'stability': KitStabilityRating.BETA},
			{'name': 'perl-kit', 'branch': '5.24-prime', 'source': 'funtoo_prime_perl', 'default': True},
			{'name': 'perl-kit', 'branch': '5.26-prime', 'source': 'funtoo_mk3_prime', 'default': False, 'stability': KitStabilityRating.DEV},
			{'name': 'python-modules-kit', 'branch': 'master', 'source': 'funtoo_current', 'default': True, 'stability': KitStabilityRating.PRIME},

Here, we see the beginning of the definition of the KitFoundation class, which contains the kit_groups class variable. This variable, as you can see, defines a bunch of kits -- their name, the branch, a stability value that defines the stability rating of the kit, and other values. Note that if a kit is marked as default, it is assumed to have a kit stability rating of PRIME. You will also see that there is a source key, which defines the group of repositories and overlays that are used to create this kit.

Here are three important things to take away from this part of code: First, this is the official master definition of what kits exist, their stability rating, and what overlays/repositories are used to generate each kit. The next thing to take away from this code is that it is certainly possible to modify these settings for your custom meta-repo so that they are different from the Funtoo defaults.

Here is the third and most important thing about kit_groups -- the order that they are defined determines the order which the merge scripts try to match packages. The merge-scripts/package-sets files will be processed in order that the kits are defined in kit_groups, so core-kit first, security-kit second, etc. Once a catpkg has been added to a kit, it will not be added to any successive kits.

Going Deeper into Foundations.py

Moving right along, you will see a section of the KitFoundation class that looks like this:

   foundations.py (python source code) - foundations.py python_kit_settings

python_kit_settings = {
		#	branch / primary python / alternate python / python mask (if any)
		'master': {
			"primary": "python3_6",
			"alternate": "python2_7",
			"mask": None
		},
		'3.4-prime': {
			"primary": "python3_4",
			"alternate": "python2_7",
			"mask": ">=dev-lang/python-3.5"
		},
		'3.6-prime': {
			"primary": "python3_6",
			"alternate": "python2_7",
			"mask": ">=dev-lang/python-3.7"
		},
		'3.6.3-prime': {
			"primary": "python3_6",
			"alternate": "python2_7",
			"mask": ">=dev-lang/python-3.7"
		}
	}

This section is used to define special settings for python-kit. Specifically, for each branch of python-kit, it defines the primary version of python that we will attempt to use to satisfy python-single USE dependencies, followed by an alternate value to use if the ebuild does not support the first. We can also specify a mask to be injected into each python-kit to mask certain versions of python that should be masked due to lack of support in that kit.

Foundations.py Kit Source Definitons

Next, we have a section that looks like this:

   foundations.py (python source code) - foundations.py kit_source_defs

kit_source_defs = {
		"funtoo_current": [
			{"repo": "flora"},
			{"repo": "faustoo"},
			{"repo": "fusion809"},
			{"repo": "gentoo-staging"}
		],
		"funtoo_mk2_prime": [
			{"repo": "flora", },
			{"repo": "faustoo"},
			{"repo": "fusion809", "src_sha1": "489b46557d306e93e6dc58c11e7c1da52abd34b0", 'date': '31 Aug 2017'},
			{"repo": "gentoo-staging", "src_sha1": '80d2f3782e7f351855664919d679e94a95793a06', 'date': '31 Aug 2017'},
			# add current gentoo-staging to catch any new ebuilds that are not yet in our snapshot above (dev-foo/* match)
			{"repo": "gentoo-staging-underlay"},
		],
		"funtoo_mk3_prime": [
			{"repo": "flora", },
			{"repo": "faustoo", },
			{"repo": "fusion809", "src_sha1": "8733034816d3932486cb593db2dfbfbc7577e28b", 'date': '09 Oct 2017'},
			{"repo": "gentoo-staging", "src_sha1": '2de4b388863ab0dbbd291422aa556c9de646f1ff', 'date': '10 Oct 2017'},
			{"repo": "gentoo-staging-underlay"},
		], ...

This section contains definitions of various overlay stacks, called "kit sources". Kit sources are a combination of overlays, arranged in a python list ([ ]). A kit source serves as a unified collection of source catpkgs for a particular kit. Each kit can have one kit source. Kit sources may be shared among kits for consistency purposes, and to avoid duplication and to help organization. Note that this is where we specify branch or SHA1 that is used for each overlay. If no SHA1 is specified, the merge scripts will use the top commit of master as the source for that overlay (ie. the overlay will track upstream.)

   Note

A "catpkg" is a category/package combination, like sys-apps/portage, with no version information. When we match catpkgs, we look for a cat/pkg directory in the overlay, and if we find a match, we grab the full contents of the catpkg directory from the overlay, and ignore any other occurrence of the same catpkg in other overlays. When we talk of "package-set" rules below, we are referring to text files that define what catpkgs go into what kits. We can specify catpkgs literally or use patterns and other approaches to select catpkgs destined for a particular kit. The merge scripts uses these package-set rules files to define what goes in each kit. But where they search for each catpkg is defined in foundations.py, above.

Currently, package-set rules (see note above) are applied in the same order that the overlay appears in the kit_source_defs list -- so for "funtoo_current", package-set rules -- an attempt to match catpkg patterns for a kit -- will be applied to flora first, then faustoo, then fusion809, etc, with gentoo-staging searched last. In theory, this would allow any "upper" overlay to override catpkgs in later-appearing overlays. However, this is not the case, because the "upper" overlays are generally "locked down" so that they are only allowed to provide a particular set of catpkgs, or are allowed to provide any catpkg as long as it doesn't appear in Gentoo, for example. So in actual practice, gentoo-staging is set up to have the higher priority, even though it is searched for matches later. Also notice that newer kit source definitions now have a "bottom" gentoo-staging-underlay definition that has a special purpose -- to allow us to grab new Gentoo catpkgs that aren't in our snapshotted gentoo-staging repository yet. This works because gentoo-staging-underlay tracks upstream master, while gentoo-staging is locked to a particular point in the past. This way we get brand-new gentoo catpkgs, but existing gentoo catpkgs stay locked to a particular point in time.

Completing the Deep Dive

Our deep dive into the functionality of foundations.py is almost complete -- just one more part to go! You will notice the following code at the end of the file:

   foundations.py (python source code) - foundations.py overlay definitions

@property
	def overlays(self):
		return {
			# use gentoo-staging-2017 dirname to avoid conflicts with ports-2012 generation
			"gentoo-staging": {"url": self.config.gentoo_staging, "dirname": "gentoo-staging-2017"},
			"gentoo-staging-underlay": {"url": self.config.gentoo_staging, "dirname": "gentoo-staging-2017-underlay"},
			"faustoo": {"url": "https://github.com/fmoro/faustoo.git", "eclasses": [
				"waf",
				"googlecode"
			],
			            # SKIP any catpkgs that also exist in gentoo-staging (like nvidia-drivers). All others will be copied.
			            "filter": ["gentoo-staging"],
			            # well, I lied. There are some catpkgs that exist in gentoo-staging that we DO want to copy. These are the
			            # ones we will copy. We need to specify each one. This list may change over time as faustoo/gentoo gets stale.
			            "force": [
				            "dev-java/maven-bin",
				            "dev-java/sun-java3d-bin",
				            "dev-php/pecl-mongo",
				            "dev-php/pecl-mongodb",
				            "dev-python/mongoengine",
				            "dev-python/pymongo",
				            "dev-util/idea-community",
				            "dev-util/webstorm",
				            "x11-wm/blackbox"
			            ]
			            },
			"fusion809": {"url": "https://github.com/fusion809/fusion809-overlay.git", "select": [
				"app-editors/atom-bin",
				"app-editors/notepadqq",
				"app-editors/bluefish",
				"app-editors/textadept",
				"app-editors/scite",
				"app-editors/gvim",
				"app-editors/vim",
				"app-editors/vim-core",
				"app-editors/sublime-text"
			]
			              },  # FL-3633, FL-3663, FL-3776
			"plex": {"url": "https://github.com/Ghent/funtoo-plex.git", "select": [
				"media-tv/plex-media-server",
			],
			         },
			# damex's deadbeef (music player like foobar2000) overlay
			"deadbeef": {"url": "https://github.com/damex/deadbeef-overlay.git", "copyfiles": {
				"profiles/package.mask": "profiles/package.mask/deadbeef.mask"
			},
			             },
			# damex's wmfs (window manager from scratch) overlay
			"wmfs": {"url": "https://github.com/damex/wmfs-overlay.git", "copyfiles": {
				"profiles/package.mask": "profiles/package.mask/wmfs.mask"
			},
			         },
			"flora": {"url": self.config.flora, "copyfiles": {
				"licenses/renoise-EULA": "licenses/renoise-EULA"
			},
			          },
		}

This section of foundations.py declares the names for the actual overlays themselves, where we can clone them from, and also helps us answer the question of which overlay has priority over another. In the case of the fusion809 overlay, for example, we will only grab a select set of catpkgs, specified in the select list. Whereas in the faustoo overlay, we will force the merge scripts to grab updates the the catpkgs listed in the force list, but skip any catpkgs in faustoo that also appear in our gentoo-staging snapshot. All these special rules define exceptions to the normal processing logic of the merge scripts, which will look for catpkg matches in the first overlay listed in the kit source definition, before proceeding to the second, etc.

Wrapping Up Foundations.py

We certainly have a lot of levers we can use to control what gets included in a particular kit. Using foundations.py, we can add or remove overlays we want to use, and we can also alter the order they are scanned by the merge scripts. We can control what snapshots are used, and vary this on a per-kit basis. And in the overlay definitions, we can also create exceptions to the normal processing order, to effectively prioritize gentoo catpkgs over overlay catpkgs, even if the overlays are processed first. Or we can limit the scope of what catpkgs we scan a particular overlay for, or force certain catpkgs to be copied from a particular overlay if we know we want those particular ones.

All that is left to cover are the package-set rules, which define which catpkgs go into each kit, and kit-fixups, which allow us to have Funtoo overrides for any catpkgs we want. We'll cover those topics next.

Package Sets

Package sets define which catpkgs go in which kits. The package set files are located at kit-fixups/package-sets. They can either consist of a single file named (kit)-packages or a directory with the same name. In the case of a directory, all of the files inside the directory are concatenated and used as a package set.

Package Set Format

Package sets are a text-based format that consist of one catpkg entry per line. Let's look at the various types of package sets entries:

Entry type	Example	Explanation
literal	sys-apps/portage	Specify a single catpkg by exact name.
literal with move	sys-apps/oldpkg -> sys-foo/newpkg	Specify a catpkg by exact name, but copy it over as a new name.
category wildcard	sys-apps/*	Specify all packages that appear in a particular category.
category wildcard with exceptions	sys-apps/* -sys-apps/foo -sys-apps/bar	Specify all packages that appear in a particular category, with some exceptions.
regex	sys-.*/foo.*	Specify all catpkgs that match a particular regex.
dependencies in category	@depsincat@:x11-base/xorg-x11:media-fonts	In this example, anything in the media-fonts category that has dependencies upon x11-base/xorg-x11.
maintainer	@maintainer@:dev-lang:ml@gentoo.org	In this example, all dev-lang packages that have a maintainer of ml@gentoo.org.
has eclass	@has_eclass@:kde5	In this example, all catpkgs that use an eclass of kde5.eclass.
category has eclass	@cat_has_eclass@:x11-apps:xorg-2	In this example, all catpkgs in the x11-apps category that use an eclass of xorg-2.eclass.

   Important

It is possible to blacklist certain catpkgs for inclusion in particular kit by creating a "skip" list at kit-fixups/package-sets/mykitnamehere-packages-skip. This file can contain individual catpkgs, one per line, that should not be included in the package-set, even if they match any rules above. Note that you must use literal catpkgs in the skip list -- no patterns or other special matches are supported.

Package Moves

   Note

The "literal with move" option is a new feature, described below.

The "literal with move" package set syntax described above is one way to tell the merge scripts to copy a catpkg from a source location but give it a new catpkg name.

If the old package name is found, it will be copied over as the new name. If the old name is not found, but the new name is found, the new name will be copied over as the new name.

You can also specify package moves by creating a file called kit-fixups/move-maps/kitname containing the same "literal with move" syntax. Global move maps can be placed in kit-fixups/move-maps/global. Also note that kit-fixups/move-maps/nokit or kit-fixups/move-maps/global is the only way to perform funtoo package moves for nokit. I recommend using the "global" method since it will still automatically work if someone else moves your package into a kit.}}

The difference between the "inline" (inside a package-set) and the "out-of-band" (in the move-maps/ directory method is that the first method adds both the old and the new catpkg name to the package set. The second method provides package move information that will apply only if the catpkg happens to be part of the package-set.

If you need to just rename a single package in a kit, it's fine to use the inline method. For more capability, move-maps/ are your friend. They're more powerful because a move-map specified with move-maps/ will also apply to any special wildcard package-set matches via @regex@, etc, whereas the inline method is limited by design. Also, the out-of-band method allows you to perform package moves on catpkgs in nokit.

Package Sets -- Putting It All Together

Here are some important facts about package sets:

1. Package set matches are executed in a particular order, and this order is defined by the order of kits in kit_groups in foundations.py.
2. Once a catpkg is matched during processing of a kit, that catpkg is assigned to that kit, and cannot appear in another kit.
3. If a catpkg is included in a particular branch of a kit, then that catpkg will appear in all branches of that kit, assuming it is available.
4. Each kit and branch defined in kit_groups specifies a "source" -- an entry in kit_source_defs which in turn defines a stack of repositories/overlays and associated SHA1 commits to use as sources for catpkgs.

So, here's how the package set processing would begin. If we look at kit_groups in foundations.py, we see that core-kit 1.0-prime is listed first. So we will look for catpkg matches for core-kit 1.0-prime using the kit-fixups/package-sets/core-kit-packages package set directory. We will apply these match rules against the gentoo_prime_protected kit source definition, and we will look for matches in each repository in the order listed in the kit_source_defs entry. The first match we find will be used as the source catpkg. But remember that we have specific rules in place, defined in the overlays property, that effectively gives gentoo-staging priority for most of the catpkgs.

After this is done, we will then process core-kit 1.2-prime, since 1.1-prime is deprecated and will be skipped, and then continue to core-hw-kit, and continue to work down the kit_groups list. This process will build up a set of catpkgs that will appear in each kit.

Kit Fixups

And finally, we have saved a very key part of the kit generation process for last. The kit-fixups repository is so named because it contains fixups, which are forked Funtoo catpkgs that are used to override catpkgs that appear in the upstream overlays and repositories. They have a special structure. We will look at the structure of the core-kit fixup directory, although others will follow the same model:

CatPkg Path	Description
kit-fixups/core-kit/global/sys-apps/portage	Due to the global directory, this catpkg will always be used when a package set specifies a match for sys-apps/portage, for all branches of core-kit.
kit-fixups/core-kit/curated/sys-apps/portage	Due to the curated directory, this catpkg will always be used when a package set specifies a match for sys-apps/portage, for all branches of core-kit except a master branch.
kit-fixups/core-kit/1.2-prime/sys-apps/portage	Due to the 1.2-prime directory, this catpkg will always be used when a package set specifies a match for sys-apps/portage, for the 1.2-prime branch of core-kit only.

Remember that kit-fixups is designed so that a fixup will always override any upstream packages. This makes it easy to keep track of Funtoo-maintained core packages. And also note that the flora repository should be used for "bonus" packages while kit-fixups should focus more on forks of critical system packages and bug fixes for Funtoo. This way, we can keep contributed ebuilds separate from core operating system ebuilds and associated bug fixes for upstream issues.

Developer Q&A

This section contains various tasks that a developer may need to perform, and what steps should be taken to perform each of these steps.

I want to move a catpkg sys-apps/blah from core-kit to foobar-kit.

To do this, first we'll note that core-kit comes before foobar-kit in kit_groups. This means that core-kit's package set rules will run first. So we will want to make sure that sys-apps/blah does not match any rules in the core-kit package-set. This can be done by possibly removing a package-set rule, or using a wildcard with exclusion like sys-apps/* -sys-apps/blah. If this doesn't work, a file can be created called core-kit-packages-skip which contains exclusions, and sys-apps/blah can be added to a line in this file. Then, you will want to make sure that sys-apps/blah does match a package set rule for foobar-kit.

I want to move a catpkg sys-apps/blah from foobar-kit to core-kit.

To do this, first we'll note that foobar-kit comes after core-kit in kit-groups, so core-kit's package set rules will run first. We can thus simply add something that will match 'sys-apps/blah' to core-kit's package-set rules. Once sys-apps/blah is included in core-kit, it will not be available for inclusion in foobar-kit, even if it has an identical rule, or a rule like 'sys-apps/*'. However, note that it is good practice to clean up any rules in foobar-kit that you know are no longer matching any catpkgs.

   Note

The above two approaches can be used to move catpkgs between kits transparently to the end-user. In the next ego sync, the catpkg will atomically move from one kit to another and no re-emerging will be required, even if the user had emerged the package from the 'old' kit location.

I want to contribute a cool package to Funtoo.

To do this, you will want to open a pull request against https://github.com/funtoo/flora. Flora is used for all 'bonus' community-contributed ebuilds.

I want to fix a bug in a particular ebuild.

To do this, first find out where the ebuild is coming from. A good way to do this is to type ls -d /var/git/meta-repo/kits/*/sys-apps/foobar, which will show you what kit it is in. Running emerge -s sys-apps/foobar will also display this information. For research purposes, it is often useful to find where the original catpkg was sourced from. You can consult https://ports.funtoo.org/packages.xml which contains a list of all catpkgs and their source repository. After doing some initial research and seeing what's wrong, you might have a fix for the ebuild. Generally, the best way to fix the ebuild is to clone https://github.com/funtoo/kit-fixups and create an appropriate fixup for the ebuild if none exists, and simply improve our fixup if one exists already. Then you can create a GitHub pull request, or open a bug on bugs.funtoo.org, or both. Remember that fixup catpkgs will totally replace all upstream ebuilds, so you may need to include multiple versions of the ebuild, even ones that don't need a fix, if they are still needed for certain packages.

   Note

If you want to fix a bug in an ebuild and you find that the ebuild comes from flora, you will want to fork flora and submit a pull request against flora instead.

I want to make a particular branch of a kit the default kit.

To do this, you will modify kit_groups and set the kit you want to be default to have 'default' : True or 'stability' : KitStabilityRating.PRIME, or both. Only one kit branch can be set as default.

I don't want to generate a particular branch of a kit.

To prevent a branch of a kit from being generated, set its stability to KitStabilityRating.DEPRECATED in kit_groups.

I want to generate a new kit branch that uses much newer ebuilds from Gentoo or from an upstream repo.

First, define a new entry in kit_source_defs that contains the collection of overlays and repos you want to use as sources. Specify the SHA1 commits you want to use for each repo (or don't specify one to use master.) Then, you will want to add a new kit definition to kit_groups, in the "prime" section.

I want to include a package in Funtoo, but move it to a new name.

To do this, use the "literal with package move" format in the package set (see section on #Package Set Format, above). You can also use files in kit_fixups/move-maps/kitname or kit_fixups/move-maps/global (info in a note below the Package Set syntax section.).

   Important

Be sure to stop by #funtoo-dev on irc.freenode.net if you need further assistance! We are here to help.