注意：

The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.

Difference between revisions of "Funtoo:Metatools/Advanced Usage"

From Funtoo
Jump to navigation Jump to search
 
(27 intermediate revisions by 3 users not shown)
Line 1: Line 1:
{{Subpages|Gitolite Setup}}
{{Subpages|Gitolite Setup}}
{{Important|These pages are out of date. Please see the latest [[Funtoo:Metatools]] page for usage information.}}


== Overview ==
== Overview ==
Line 5: Line 7:
Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.
Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.


Required for merge scripts is  {{c|dev-python/lxml}}.
Before starting, please read and understand the following pages:
* [[Kit-fixups]]
* [[Kit-fixups/FAQ]]


== The Code ==
== Different Approaches ==


You can find the code that does this on code.funtoo.org, housed at https://code.funtoo.org/bitbucket/projects/CORE/repos/merge-scripts/browse. The script that does all the heavy-lifting is called {{c|merge-all-kits}}. Let's clone it from git, on the machine that will be generating new kits and meta-repo:
{{c|merge-all-kits}} can be used in a variety of ways. If you are an individual developer, you may want to use {{c|merge-all-kits}} in ''developer mode'', which is its default mode. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.
 
In developer mode, a {{c|meta-repo}} and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history.
 
Another way to use {{c|merge-all-kits}} is with {{c|gitolite}} as a remote, which is documented under the [[#Gitolite Setup|Gitolite Setup]] sub-page. Using this configuration, {{c|merge-all-kits}} will write all of its destination git repositories to gitolite. You can then configure other systems to {{c|ego sync}} from gitolite as a source. This can be good in a distributed development environment. This method can also be used to create a meta-repo and kits that ''diverge'' from Funtoo's kit but still connect to its history.
 
Finally, the {{c|merge-all-kits}} script can be run in a production mode, which is very similar to using it with gitolite except that you may be pushing up your destination meta-repo and kits to a public location like GitHub or GitLab.
 
When first starting out with {{c|merge-all-kits}}, you should definitely start with developer mode and see if that meets your needs first.
 
== Getting The Code ==
 
You can find the code that does this on code.funtoo.org, housed at [https://code.funtoo.org/bitbucket/projects/CORE/repos/merge-scripts/browse code.funtoo.org]. The script that does all the heavy-lifting is called {{c|merge-all-kits}}. Let's clone it from git, on the machine that will be generating new kits and meta-repo:


{{console|body=
{{console|body=
Line 15: Line 31:
}}
}}


== Different Approaches ==
You will also want to install the following dependencies so that the code can run:


{{c|merge-all-kits}} can be used in a variety of ways. You can use the script to generate your own dynamically-generated kits and meta-repo -- this is the default behavior when you run the script. In this mode, a {{c|meta-repo}} and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk, and they will not have any history shared with the Funtoo project.
{{console|body=
# ##i##emerge jinja lxml
}}


Another way to use {{c|merge-all-kits}} is with {{c|gitolite}}, which is documented under the [[#Gitolite Setup|Gitolite Setup]] sub-page. Using this configuration, {{c|merge-all-kits}} will write all of its destination git repositories to gitolite.
== Configuration and Paths ==


== Configuration ==
In general, you should run {{c|merge-all-kits}} as a regular user, and this user needs to be a member of the {{c|portage}} group.


Now that merge-scripts is cloned, we will need to create a {{c|~/.merge}} configuration file. Here is an example file that can serve as a starting point. In this example, developer {{c|bcowan}} has forked the kit-fixups repository on code.funtoo.org, and wants to test his personal changes by generating a complete meta-repo of his own:
In the user's home directory, create a {{f|~/.merge}} directory with the following contents:


{{file|name=/root/.merge|body=
{{file|name=/root/.merge|body=
Line 29: Line 47:


flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
kit-fixups = ssh://git@code.funtoo.org:7999/~bcowan/kit-fixups.git
kit-fixups = https://code.funtoo.org/bitbucket/scm/co/flora.git
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git
[destinations]
base_url = repos@repohost:wildrepo/staging


[branches]
[branches]
Line 43: Line 57:
}}
}}


==== Sources Section ====
By default, {{c|merge-all-kits}} will create a {{c|~/repo_tmp}} directory where it will store all of its work. Inside {{f|~/repo_tmp/source-trees}}, source git trees that will be used to generate meta-repo will be cloned. Inside {{f|~/repo_tmp/dest-trees}}, git repositories will be created to write results into. The meta-repo will be stored at {{c|/~repo_temp/dest-trees/meta-repo}}, and in developer mode, the kits will be stored in meta-repo, at {{c|~/repo_tmp/dest-trees/meta-repo/kits}}, so that meta-repo is configured 'ready to use' at it would appear at {{f|/var/git/meta-repo}}. When 'production' mode is specified via the {{c|--prod}} switch, the meta-repo will not have kits stacked inside it but instead the kits will be written directly into {{c|~/repo_tmp/dest-trees}}.
 
Let's walk through this configuration file. The {{c|[sources]}} section defines locations of repositories that the merge scripts will use as sources for creating kits and meta-repo. In the above sample config, we are using the official [[Flora]] repository from Funtoo, and the official gentoo-staging repository used by Funtoo, but we are using our own fork of [https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse kit-fixups], which will allow us to add new ebuilds that will appear in kits, such as bug fixes to existing ebuilds in kits, as well as security fixes. For a core Funtoo Linux developer, this is a good way to start. If you are more interested in contributing third-party ebuilds, then you may instead choose to create your own fork of [https://code.funtoo.org/bitbucket/projects/CO/repos/flora/browse flora], and use our standard kit-fixups repository. Or, you could choose to create forks of both. The recommended best practice is to use our upstream repos when possible, and fork only those repos you want to customize. This way, you'll ensure that you have the most up-to-date versions of ebuilds in those unforked repos.
 
==== Branches Section ====
 
The {{c|[branches]}} section is used to define the default branches that are used by the merge-scripts. In general, sticking with {{c|master}} is fine, but if you need the flexibility, you can point the merge scripts to a particular feature branch to use instead, for example.


==== Work Section ====
A metadata cache will also be created and stored at {{c|~/repo_tmp/metadata-cache}}, and your user must be a member of the {{c|portage}} group for {{c|merge-all-scripts}} to be able to properly generate the metadata cache.


The {{c|[work]}} section is used to define paths where the merge-scripts will do their work. The {{c|source}} and {{c|destination}} settings above are good defaults, and define where the merge-scripts will clone source repositories and destination (written to) repositories, such as kits and meta-repo. These are kept in two separate hierarchies so they don't get mixed up.
== Running Merge-All-Kits ==


== Generating New Kits ==
To run {{c|merge-all-kits}}, use it as follows:
 
With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be stored on '''repohost'''. Here are the steps you'd perform:


{{console|body=
{{console|body=
 
$ ##i##path/to/merge-scripts/bin/merge-all-kits 1.4-release
# ##i##cd /root/merge-scripts
# ##i##bin/merge-all-kits 1.3-release
}}
 
Before starting the script for the first time, you should emerge jinja and configure your git user.name and user.email variables.
 
{{console|body=
# ##i##emerge -u jinja
# ##i##git config --global user.email "you@example.com"
# ##i##git config --global user.name "Your Name"
}}
}}


{{c|merge-all-kits}} will proceed to create new kits and meta-repo, and will push them up to repos@repohost:wildrepo/staging/meta-repo, repos@repohost:wildrepo/staging/core-kit, etc. This process can take quite a while but has been optimized to run quickly on multi-core systems.
The script will then run, and it will git clone flora, kit-fixups and gentoo-staging from the URIs in the {{c|~/.merge}} file, and assemble them into a complete meta-repo. The meta-repo will be located at {{f|$HOME/repo_tmp/dest-trees/meta-repo}}.
 
== Using New Kits ==


Now that the new meta-repo and kits are created, here's how you'll use them on an existing Funtoo system, instead of your official Funtoo meta-repo and kits. First, we'll want to modify {{c|/etc/ego.conf}} as follows:
To use the {{c|meta-repo}} for testing, you can add the following lines to {{f|/etc/ego.conf}}:


{{file|name=/etc/ego.conf|body=
{{file|name=/etc/ego.conf|body=
[global]
[global]
 
meta_repo_path = /home/user/repo_tmp/dest-trees/meta-repo
sync_user = root
sync_base_url = repos@repohost:wildrepo/staging/{repo}
 
# Yes, you are supposed to have a literal "{repo}", above. Ego recognizes this special pattern.
 
# You can have whatever [kits] section you want, below...
}}
 
You will want to make sure that whatever system is connecting to '''repohost''' is permitted by gitolite, and has its {{c|/root/.ssh/id_rsa.pub file}} stored and committed to the gitolite-admin repository's {{c|keydir/}} and is referenced in {{c|conf/gitolite.conf}}. Otherwise, gitolite will not allow this system to connect. Of course, if you are configuring this on the system that's running {{c|merge-all-kits.py}}, it already has RW permissions to these repositories. Otherwise, you will want to make sure that gitolite has the system's keys as part of its {{c|@reporead}} group.
 
{{Note|We change the {{c|sync_user}} to {{c|root}} to force {{c|ego sync}} to use {{c|/root/.ssh/id_rsa}} for authentication. By default, {{c|ego sync}} will attempt to use the {{c|portage}} user which does not have a private key installed, and thus will not be able to authenticate with gitolite. Rather than mess with the {{c|portage}} user and give it a proper home directory and ssh key pair, it's easier just to not drop perms to {{c|portage}} in the first place.}}
 
Now, let's move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run {{c|ego sync}}:
 
{{console|body=
# ##i##cd /var/git
# ##i##mv meta-repo meta-repo.official
# ##i##ego sync
Syncing xorg-kit branch 1.20-release
Cloning into '/var/git/meta-repo/kits/xorg-kit'...
Initialized empty Git repository in /home/repos/repositories/wildrepo/staging/xorg-kit.git/
fatal: Remote branch 1.20-release not found in upstream origin
##r##ERROR: Could not clone kit 'xorg-kit' into '/var/git/meta-repo/kits/xorg-kit'.
fatal: the remote end hung up unexpectedly
}}
 
Ego will now sync your custom repository, ''but it will fail.''. This is OK -- it is failing because gitolite does not have local copies of independently-maintained (ie. not auto-generated) kits. To fix this, you can either create mirrors of the independent kits on your gitolite, or alternatively perform this simple work-around. Go to https://code.funtoo.org/bitbucket/projects/INDY and for each kit listed there, clone to {{f|/var/git/meta-repo/kits}} as root, as follows:
 
{{console|body=
# ##i##cd /var/git/meta-repo/kits
# ##i## git clone https://code.funtoo.org/bitbucket/scm/indy/gnome-kit.git
# ##i## git clone https://code.funtoo.org/bitbucket/scm/indy/python-kit.git
... etc ...
}}
}}


Once this is complete, try an {{c|ego sync}} again, and with local copies of the independent kits it should all work:
Then, using ego 2.8.0 or later, run the following command:


{{console|body=
{{console|body=
# ##i##ego sync
# ##i##ego sync --in-place
...
##g##Sync successful and kits in alignment! :)
Updating /etc/portage/repos.conf...
Updating profiles at /etc/portage/make.profile/parent...
}}
}}


If you type {{c|emerge -auDN @world}}, ego will now be using your custom kits, rather than the official Funtoo ones. This means that you can perform a variety of things you couldn't before. You can now add your own custom ebuilds to your fork of {{c|kit-fixups}}, and {{c|merge-all-kits.py}} will automatically incorporate these changes into your own custom kits. This will allow you to locally test any changes before submitting them as pull requests to Funtoo. You will also be able to maintain your own meta-repo and kits with your own local modifications, and have your systems use these meta-repo/kits instead of the official Funtoo ones.
This will ensure that the proper kit branches in your meta-repo are checked out and that all the repos.conf and profiles files in {{c|/etc/portage}} are configured correctly for your repository to be used. Note that no actual "{{c|git pull}}" will be performed -- this is intentional and a special feature of the {{c|--in-place}} option -- because your local git repositories have no remotes to pull from and contain all the information they need to be used.


== Using New Kits with Metro ==
You're now ready to use your meta-repo directly:
 
Rather than using new kits on a local Funtoo Linux system, you may want to use the kits to perform a metro build. This is desirable when you may want to use metro to perform [[Wikipedia:Continuous integration|continuous integration]] by attempting to perform a full stage1, 2 and 3 build, which is what the steps in this example accomplish. To do this, first set up metro on your system:


{{console|body=
{{console|body=
# ##i##cd /root
# ##i##emerge foo
# ##i##git clone https://code.funtoo.org/bitbucket/scm/core/metro.git
# ##i##cd metro
# ##i##scripts/autosetup
}}
Now, we will set the {{c|ego}}, administration tool of Funtoo/Linux. The way it is used with metro is independent from {{c|app-admin/ego}} installed on your box. Setup is easy as follows:
{{console|body=
# ##i##cd /root
# ##i##git clone https://code.funtoo.org/bitbucket/scm/core/ego.git
}}
This way you will have {{c|/root/ego}} directory with {{c|ego}} binary that is then used by metro.
 
After following the interactive prompts, you'll then want to perform the following steps to install an ego.conf for use by metro:
 
{{console|body=
# ##i##install -d /etc/metro/chroot/etc
# ##i##vim /etc/metro/chroot/etc/ego.conf
}}
 
To ensure that ego uses your custom local meta-repo, you'll want to ensure that the following lines are in your /etc/metro/chroot/etc/ego.conf:
 
{{file|body=
[global]
 
sync_user = root
sync_base_url = repos@repohost:wildrepo/staging/{repo}
}}
 
Without these ego options, metro will use the official meta-repo instead of your local custom meta-repo. Of course, be sure to specify any custom kit branches to use as well.
 
Once your ego.conf is installed, you can perform a metro build with something similar to the following commands:
 
{{console|body=
# ##i##cd /root/metro
# ##i##scripts/ezbuild.sh funtoo-current x86-64bit intel64-haswell full
}}
 
For a full continuous integration loop, which includes regeneration of new meta-repo contents, the following script can be used:
 
{{file|lang=bash|body=
#!/bin/bash
/root/merge-scripts/bin/merge-all-kits push
rm /home/mirror/funtoo/funtoo-current/snapshots/portage-*
/root/metro/scripts/ezbuild.sh funtoo-current x86-64bit intel64-haswell full
}}
 
The extra "rm" command is included to clean out any previous daily snapshots, to force metro to regenerate snapshots. Otherwise, any existing daily portage snapshots will be used and new ones will not be generated.
 
{{Important|If you are "switching back and forth" between different versions of kits, it may be necessary to wipe metro's binary package cache to avoid build failures caused by emerge trying to install binary packages built for a different environment. A symptom of a package cache needing to be wiped out are builds failing because they depend on a masked slot/subslot combination (and emerge asking you to remove a mask.) In this case, you will want to recursively remove the contents of {{f|/var/tmp/metro/cache/package-cache}}. For periodic updates of a consistent set of kits, this is typically not necessary.}}
 
== Changing Repo Definitions ==
 
Now that you have generated your first meta-repo, let's look at the key file that defines what overlays and repositories are used to create meta-repo, as well as what kit branches exist, and which ones are prime and which are not. You will want to turn your attention to the {{f|kit-fixups/modules/fixups/foundations.py}} file, which [https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse/modules/fixups/foundations.py can be viewed here on code.funtoo.org]]. Let's take a look at various sections of this file:
 
{{file|name=foundations.py|lang=python|desc=Top of foundations.py file|body=
#!/usr/bin/python3
 
from enum import Enum
 
 
class KitStabilityRating(Enum):
PRIME = 0  # Kit is enterprise-quality
NEAR_PRIME = 1  # Kit is approaching enterprise-quality
BETA = 2  # Kit is in beta
ALPHA = 3  # Kit is in alpha
DEV = 4  # Kit is newly created and in active development
CURRENT = 10  # Kit follows Gentoo currrent
DEPRECATED = 11  # Kit is deprecated/retired
 
 
def KitRatingString(kit_enum):
if kit_enum is KitStabilityRating.PRIME:
return "prime"
elif kit_enum is KitStabilityRating.NEAR_PRIME:
return "near-prime"
elif kit_enum is KitStabilityRating.BETA:
return "beta"
elif kit_enum is KitStabilityRating.ALPHA:
return "alpha"
elif kit_enum is KitStabilityRating.DEV:
return "dev"
elif kit_enum is KitStabilityRating.CURRENT:
return "current"
elif kit_enum is KitStabilityRating.DEPRECATED:
return "deprecated"
}}
}}
At the top of the file, we define an [[Wikipedia:Enumeration|enumeration]] of the different levels of stability that can exist for a particular kit. Kits marked with a stability rating of {{c|KitStabilityRating.PRIME}} will be tagged in the meta-repo JSON as being a "prime" kit and suitable for production use.
As you scroll down further, you will see some code like this:
{{file|name=foundations.py|lang=python|desc=foundations.py kit_groups|body=
class KitFoundation:
kit_groups = {
'prime': [
{'name': 'core-kit', 'branch': '1.0-prime', 'source': 'gentoo_prime_protected', 'default': True},
{'name': 'core-kit', 'branch': '1.1-prime', 'source': 'gentoo_prime_mk3_protected', 'stability': KitStabilityRating.DEPRECATED},
{'name': 'core-kit', 'branch': '1.2-prime', 'source': 'gentoo_prime_mk4_protected', 'stability': KitStabilityRating.BETA},
{'name': 'core-hw-kit', 'branch': 'master', 'source': 'funtoo_current', 'default': True},
{'name': 'security-kit', 'branch': '1.0-prime', 'source': 'gentoo_prime_protected', 'default': True},
{'name': 'security-kit', 'branch': '1.1-prime', 'source': 'gentoo_prime_mk3_protected', 'stability': KitStabilityRating.DEPRECATED},
{'name': 'security-kit', 'branch': '1.2-prime', 'source': 'gentoo_prime_mk4_protected', 'stability': KitStabilityRating.BETA},
{'name': 'xorg-kit', 'branch': '1.17-prime', 'source': 'funtoo_prime_xorg', 'default': False, 'stability': KitStabilityRating.PRIME},
{'name': 'xorg-kit', 'branch': '1.19-prime', 'source': 'funtoo_mk2_prime', 'default': True, 'stability': KitStabilityRating.PRIME},  # MK2
{'name': 'gnome-kit', 'branch': '3.20-prime', 'source': 'funtoo_prime_gnome', 'default': True},
{'name': 'gnome-kit', 'branch': '3.26-prime', 'source': 'funtoo_mk4_prime', 'default': False, 'stability': KitStabilityRating.DEV},
{'name': 'kde-kit', 'branch': '5.10-prime', 'source': 'funtoo_mk3_prime', 'default': False, 'stability': KitStabilityRating.DEPRECATED},
{'name': 'kde-kit', 'branch': '5.11-prime', 'source': 'funtoo_prime_kde', 'stability': KitStabilityRating.DEPRECATED},
{'name': 'kde-kit', 'branch': '5.12-prime', 'source': 'funtoo_prime_kde_late', 'default': True, 'stability': KitStabilityRating.PRIME},
{'name': 'media-kit', 'branch': '1.0-prime', 'source': 'funtoo_prime_media', 'default': False, 'stability': KitStabilityRating.DEPRECATED},
{'name': 'media-kit', 'branch': '1.1-prime', 'source': 'funtoo_mk3_prime', 'default': True, 'stability': KitStabilityRating.PRIME},  # MK3
{'name': 'media-kit', 'branch': '1.2-prime', 'source': 'funtoo_mk4_prime', 'stability': KitStabilityRating.BETA},
{'name': 'perl-kit', 'branch': '5.24-prime', 'source': 'funtoo_prime_perl', 'default': True},
{'name': 'perl-kit', 'branch': '5.26-prime', 'source': 'funtoo_mk3_prime', 'default': False, 'stability': KitStabilityRating.DEV},
{'name': 'python-modules-kit', 'branch': 'master', 'source': 'funtoo_current', 'default': True, 'stability': KitStabilityRating.PRIME},
}}
Here, we see the beginning of the definition of the {{c|KitFoundation}} class, which contains the {{c|kit_groups}} class variable. This variable, as you can see, defines a bunch of kits -- their name, the branch, a {{c|stability}} value that defines the stability rating of the kit, and other values. Note that if a kit is marked as {{c|default}}, it is assumed to have a kit stability rating of PRIME. You will also see that there is a {{c|source}} key, which defines the group of repositories and overlays that are used to create this kit.
Here are three important things to take away from this part of code: First, '''this is the official master definition of what kits exist, their stability rating, and what overlays/repositories are used to generate each kit.''' The next thing to take away from this code is that it is certainly possible to modify these settings for your custom meta-repo so that they are different from the Funtoo defaults.
Here is the third and most important thing about {{c|kit_groups}} -- the order that they are defined determines the order which the merge scripts try to match packages. The {{f|merge-scripts/package-sets}} files will be processed in order that the kits are defined in {{c|kit_groups}}, so {{c|core-kit}} first, {{c|security-kit}} second, etc. Once a catpkg has been added to a kit, it will not be added to any successive kits.
=== Going Deeper into Foundations.py ===
Moving right along, you will see a section of the {{c|KitFoundation}} class that looks like this:
{{file|name=foundations.py|lang=python|desc=foundations.py python_kit_settings|body=
python_kit_settings = {
# branch / primary python / alternate python / python mask (if any)
'master': {
"primary": "python3_6",
"alternate": "python2_7",
"mask": None
},
'3.4-prime': {
"primary": "python3_4",
"alternate": "python2_7",
"mask": ">=dev-lang/python-3.5"
},
'3.6-prime': {
"primary": "python3_6",
"alternate": "python2_7",
"mask": ">=dev-lang/python-3.7"
},
'3.6.3-prime': {
"primary": "python3_6",
"alternate": "python2_7",
"mask": ">=dev-lang/python-3.7"
}
}
}}
This section is used to define special settings for python-kit. Specifically, for each branch of python-kit, it defines the primary version of python that we will attempt to use to satisfy python-single USE dependencies, followed by an alternate value to use if the ebuild does not support the first. We can also specify a mask to be injected into each python-kit to mask certain versions of python that should be masked due to lack of support in that kit.
=== Foundations.py Kit Source Definitons ===
Next, we have a section that looks like this:
{{file|name=foundations.py|lang=python|desc=foundations.py kit_source_defs|body=
kit_source_defs = {
"funtoo_current": [
{"repo": "flora"},
{"repo": "faustoo"},
{"repo": "fusion809"},
{"repo": "gentoo-staging"}
],
"funtoo_mk2_prime": [
{"repo": "flora", },
{"repo": "faustoo"},
{"repo": "fusion809", "src_sha1": "489b46557d306e93e6dc58c11e7c1da52abd34b0", 'date': '31 Aug 2017'},
{"repo": "gentoo-staging", "src_sha1": '80d2f3782e7f351855664919d679e94a95793a06', 'date': '31 Aug 2017'},
# add current gentoo-staging to catch any new ebuilds that are not yet in our snapshot above (dev-foo/* match)
{"repo": "gentoo-staging-underlay"},
],
"funtoo_mk3_prime": [
{"repo": "flora", },
{"repo": "faustoo", },
{"repo": "fusion809", "src_sha1": "8733034816d3932486cb593db2dfbfbc7577e28b", 'date': '09 Oct 2017'},
{"repo": "gentoo-staging", "src_sha1": '2de4b388863ab0dbbd291422aa556c9de646f1ff', 'date': '10 Oct 2017'},
{"repo": "gentoo-staging-underlay"},
], ...
}}
This section contains definitions of various overlay stacks, called "kit sources". Kit sources are a combination of overlays, arranged in a python list ({{c|<nowiki>[ ]</nowiki>}}). A kit source serves as a unified collection of source catpkgs for a particular kit. Each kit can have one kit source. Kit sources ''may'' be shared among kits for consistency purposes, and to avoid duplication and to help organization. Note that this is where we specify branch or SHA1 that is used for each overlay. If no SHA1 is specified, the merge scripts will use the top commit of master as the source for that overlay (ie. the overlay will track upstream.)
{{Note|A "catpkg" is a category/package combination, like {{c|sys-apps/portage}}, with no version information. When we match catpkgs, we look for a cat/pkg directory in the overlay, and if we find a match, we grab the full contents of the catpkg directory from the overlay, and ignore any other occurrence of the same catpkg in other overlays. When we talk of "package-set" rules below, we are referring to text files that define what catpkgs go into what kits. We can specify catpkgs literally or use patterns and other approaches to select catpkgs destined for a particular kit. The merge scripts uses these package-set rules files to define what goes in each kit. But where they search for each catpkg is defined in {{f|foundations.py}}, above.}}
Currently, package-set rules (see note above) are applied in
the same order that the overlay appears in the {{c|kit_source_defs list}} -- so for "funtoo_current", package-set rules -- an attempt to match catpkg patterns for a kit -- will
be applied to flora first, then faustoo, then fusion809, etc, with gentoo-staging searched last. In theory, this would allow any "upper" overlay to override catpkgs in later-appearing overlays. However, this is not the case, because the "upper" overlays are generally "locked down" so that they are only ''allowed to provide a particular set of catpkgs'', or are allowed to provide any catpkg as long as it doesn't appear in Gentoo, for example. So in actual practice, gentoo-staging is set up to have the higher priority, even though it is searched for matches later. Also notice that newer kit source definitions now have a "bottom" gentoo-staging-underlay definition that has a special purpose -- to allow us to grab new Gentoo catpkgs that aren't in our snapshotted gentoo-staging repository yet. This works because gentoo-staging-underlay tracks upstream master, while gentoo-staging is locked to a particular point in the past. This way we get brand-new gentoo catpkgs, but existing gentoo catpkgs stay locked to a particular point in time.
=== Completing the Deep Dive ===
Our deep dive into the functionality of {{f|foundations.py}} is almost complete -- just one more part to go! You will notice the following code at the end of the file:
{{file|name=foundations.py|lang=python|desc=foundations.py overlay definitions|body=
@property
def overlays(self):
return {
# use gentoo-staging-2017 dirname to avoid conflicts with ports-2012 generation
"gentoo-staging": {"url": self.config.gentoo_staging, "dirname": "gentoo-staging-2017"},
"gentoo-staging-underlay": {"url": self.config.gentoo_staging, "dirname": "gentoo-staging-2017-underlay"},
"faustoo": {"url": "https://github.com/fmoro/faustoo.git", "eclasses": [
"waf",
"googlecode"
],
            # SKIP any catpkgs that also exist in gentoo-staging (like nvidia-drivers). All others will be copied.
            "filter": ["gentoo-staging"],
            # well, I lied. There are some catpkgs that exist in gentoo-staging that we DO want to copy. These are the
            # ones we will copy. We need to specify each one. This list may change over time as faustoo/gentoo gets stale.
            "force": [
            "dev-java/maven-bin",
            "dev-java/sun-java3d-bin",
            "dev-php/pecl-mongo",
            "dev-php/pecl-mongodb",
            "dev-python/mongoengine",
            "dev-python/pymongo",
            "dev-util/idea-community",
            "dev-util/webstorm",
            "x11-wm/blackbox"
            ]
            },
"fusion809": {"url": "https://github.com/fusion809/fusion809-overlay.git", "select": [
"app-editors/atom-bin",
"app-editors/notepadqq",
"app-editors/bluefish",
"app-editors/textadept",
"app-editors/scite",
"app-editors/gvim",
"app-editors/vim",
"app-editors/vim-core",
"app-editors/sublime-text"
]
              },  # FL-3633, FL-3663, FL-3776
"plex": {"url": "https://github.com/Ghent/funtoo-plex.git", "select": [
"media-tv/plex-media-server",
],
        },
# damex's deadbeef (music player like foobar2000) overlay
"deadbeef": {"url": "https://github.com/damex/deadbeef-overlay.git", "copyfiles": {
"profiles/package.mask": "profiles/package.mask/deadbeef.mask"
},
            },
# damex's wmfs (window manager from scratch) overlay
"wmfs": {"url": "https://github.com/damex/wmfs-overlay.git", "copyfiles": {
"profiles/package.mask": "profiles/package.mask/wmfs.mask"
},
        },
"flora": {"url": self.config.flora, "copyfiles": {
"licenses/renoise-EULA": "licenses/renoise-EULA"
},
          },
}
}}
This section of {{f|foundations.py}} declares the names for the actual overlays themselves, where we can clone them from, and also helps us answer the question of which overlay has priority over another. In the case of the fusion809 overlay, for example, we will only grab a select set of catpkgs, specified in the {{c|select}} list. Whereas in the faustoo overlay, we will force the merge scripts to grab updates the the catpkgs listed in the {{c|force}} list, but ''skip'' any catpkgs in faustoo that also appear in our gentoo-staging snapshot. All these special rules define ''exceptions'' to the normal processing logic of the merge scripts, which will look for catpkg matches in the first overlay listed in the kit source definition, before proceeding to the second, etc.
=== Wrapping Up Foundations.py ===
We certainly have a lot of levers we can use to control what gets included in a particular kit. Using {{f|foundations.py}}, we can add or remove overlays we want to use, and we can also alter the order they are scanned by the merge scripts. We can control what snapshots are used, and vary this on a per-kit basis. And in the overlay definitions, we can also create exceptions to the normal processing order, to effectively prioritize gentoo catpkgs over overlay catpkgs, even if the overlays are processed first. Or we can limit the scope of what catpkgs we scan a particular overlay for, or force certain catpkgs to be copied from a particular overlay if we ''know'' we want those particular ones.
All that is left to cover are the package-set rules, which define which catpkgs go into each kit, and kit-fixups, which allow us to have Funtoo overrides for any catpkgs we want. We'll cover those topics next.
== Package Sets ==
'''Package sets''' define which catpkgs go in which kits. The package set files are located at {{c|kit-fixups/package-sets}}. They can either consist of a single file named {{c|(kit)-packages}} or a directory with the same name. In the case of a directory, all of the files inside the directory are concatenated and used as a package set.
=== Package Set Format ===
Package sets are a text-based format that consist of one catpkg entry per line. Let's look at the various types of package sets entries:
{{TableStart}}
<tr><th>Entry type</th><th>Example</th><th>Explanation</th></tr>
<tr><td>literal</td><td>{{c|sys-apps/portage}}</td><td>Specify a single catpkg by exact name.</td></tr>
<tr><td>literal with move</td><td>{{c|sys-apps/oldpkg -> sys-foo/newpkg}}<td>Specify a catpkg by exact name, but copy it over as a new name.</td></tr>
<tr><td>category wildcard</td><td>{{c|sys-apps/*}}</td><td>Specify all packages that appear in a particular category.</td></tr>
<tr><td>category wildcard with exceptions</td><td>{{c|sys-apps/* -sys-apps/foo -sys-apps/bar}}</td><td>Specify all packages that appear in a particular category, with some exceptions.</td></tr>
<tr><td>regex</td><td>{{c|sys-.*/foo.*}}</td><td>Specify all catpkgs that match a particular regex.</td></tr>
<tr><td>dependencies in category</td><td>{{c|@depsincat@:x11-base/xorg-x11:media-fonts}}</td><td>In this example, anything in the {{c|media-fonts}} category that has dependencies upon {{c|x11-base/xorg-x11}}.</td></tr>
<tr><td>maintainer</td><td>{{c|@maintainer@:dev-lang:ml@gentoo.org}}</td><td>In this example, all {{c|dev-lang}} packages that have a maintainer of {{c|ml@gentoo.org}}.</td></tr>
<tr><td>has eclass</td><td>{{c|@has_eclass@:kde5}}</td><td>In this example, all catpkgs that use an eclass of {{c|kde5.eclass}}.</td></tr>
<tr><td>category has eclass</td><td>{{c|@cat_has_eclass@:x11-apps:xorg-2}}</td><td>In this example, all catpkgs in the {{c|x11-apps}} category that use an eclass of {{c|xorg-2.eclass}}.</td></tr>
{{TableEnd}}
{{Important|It is possible to blacklist certain catpkgs for inclusion in particular kit by creating a "skip" list at {{f|kit-fixups/package-sets/mykitnamehere-packages-skip}}. This file can contain individual catpkgs, one per line, that should not be included in the package-set, even if they match any rules above. Note that you must use literal
catpkgs in the skip list -- no patterns or other special matches are supported.}}
==== Inline Package Moves ====
{{Note|The "literal with move" option is a new feature, described below.}}
The "literal with move" package set syntax described above is one way to tell the merge scripts to copy a catpkg from a source location but give it a new catpkg name, and they are specified directly in the package-set files.
If the old package name is found, it will be copied over as the new name. If the old name is not found, but the new name is found, the new name will be copied over as the new name. So either the old name or the new name, if found, will be copied over.
===== Move-Maps =====
You can also specify package moves by creating a file called {{f|kit-fixups/move-maps/kitname}} containing the same "literal with move" syntax. Global move maps can be placed in {{f|kit-fixups/move-maps/global}}. Also note that {{f|kit-fixups/move-maps/nokit}} or {{f|kit-fixups/move-maps/global}} is the only way to perform funtoo package moves for nokit. I recommend using the "global" method since it will still automatically work if someone else moves your package into a kit.
This move-maps functionality works similarly to package-moves that appear within a kit package-set, except that these moves do not automatically add either specified catpkg to the kit. So there must be something in the package-set that matches the old package name. After the old package is matched, additional logic looks at the move-maps and see if the match is in a move-map that tells us to copy it over as the "new" name. The old package will be copied over as the new name.
If you need to just rename a single package in a kit, it's fine to use the inline method. For more capability, {{f|move-maps/}} are your friend. They're more powerful because a move-map specified with {{f|move-maps/}} will also apply to any special wildcard package-set matches via {{c|@regex@}}, etc, whereas the inline method is limited by design. Also, the out-of-band method allows you to perform package moves on catpkgs in nokit.
{{Important|Package moves give you the ability to rename catpkgs as they appear in kits. We still need to add functionality to provide this data to Portage so that it can update any package database entries for packages installed under the old name. This part is not done yet, so this is considered a testing-only feature for the time being and should only be used on local kit-fixups overlays for testing, not in our official kit-fixups repo yet.}}
=== Package Sets -- Putting It All Together ===
Here are some important facts about package sets:
# Package set matches are executed in a particular order, and this order is defined by the order of kits in {{c|kit_groups}} in {{c|foundations.py}}.
# Once a catpkg is matched during processing of a kit, that catpkg is assigned to that kit, and cannot appear in another kit.
# If a catpkg is included in a particular branch of a kit, then that catpkg will appear in all branches of that kit, assuming it is available.
# Each kit and branch defined in {{c|kit_groups}} specifies a "source" -- an entry in {{c|kit_source_defs}} which in turn defines a stack of repositories/overlays and associated SHA1 commits to use as sources for catpkgs.
So, here's how the package set processing would begin. If we look at {{c|kit_groups}} in {{c|foundations.py}}, we see that core-kit 1.0-prime is listed first. So we will look for catpkg matches for core-kit 1.0-prime using the {{c|kit-fixups/package-sets/core-kit-packages}} package set directory. We will apply these match rules against the {{c|gentoo_prime_protected}} kit source definition, and we will look for matches in each repository in the order listed in the {{c|kit_source_defs}} entry. The first match we find will be used as the source catpkg. But remember that we have specific rules in place, defined in the {{c|overlays}} property, that effectively gives gentoo-staging priority for most of the catpkgs.
After this is done, we will then process core-kit 1.2-prime, since 1.1-prime is deprecated and will be skipped, and then continue to core-hw-kit, and continue to work down the {{c|kit_groups}} list. This process will build up a set of catpkgs that will appear in each kit.
== Kit Fixups ==
And finally, we have saved a very key part of the kit generation process for last. The {{c|kit-fixups}} repository is so named because it contains ''fixups'', which are forked Funtoo catpkgs that are used to override catpkgs that appear in the upstream overlays and repositories. They have a special structure. We will look at the structure of the core-kit fixup directory, although others will follow the same model:
{{TableStart}}
<tr><th>CatPkg Path</th><th>Description</th></tr>
<tr><td>{{f|kit-fixups/core-kit/global/sys-apps/portage}}</td><td>Due to the {{c|global}} directory, this catpkg will always be used when a package set specifies a match for {{c|sys-apps/portage}}, for '''all''' branches of core-kit.</td></tr>
<tr><td>{{f|kit-fixups/core-kit/curated/sys-apps/portage}}</td><td>Due to the {{c|curated}} directory, this catpkg will always be used when a package set specifies a match for {{c|sys-apps/portage}}, for '''all''' branches of core-kit '''except a master branch.'''</td></tr>
<tr><td>{{f|kit-fixups/core-kit/1.2-prime/sys-apps/portage}}</td><td>Due to the {{c|1.2-prime}} directory, this catpkg will always be used when a package set specifies a match for {{c|sys-apps/portage}}, for the '''1.2-prime''' branch of core-kit only.</td></tr>
{{TableEnd}}
Remember that kit-fixups is designed so that ''a fixup will always override any upstream packages.'' This makes it easy to keep track of Funtoo-maintained core packages. And also note that the flora repository should be used for "bonus" packages while kit-fixups should focus more on forks of critical system packages and bug fixes for Funtoo. This way, we can keep contributed ebuilds separate from core operating system ebuilds and associated bug fixes for upstream issues.
== Developer Q&A ==
This section contains various tasks that a developer may need to perform, and what steps should be taken to perform each of these steps.
; I want to move a catpkg sys-apps/blah from core-kit to foobar-kit.: To do this, first we'll note that ''core-kit comes before foobar-kit in {{c|kit_groups}}. This means that core-kit's package set rules will run first. So we will want to make sure that sys-apps/blah does '''not''' match any rules in the core-kit package-set. This can be done by possibly removing a package-set rule, or using a wildcard with exclusion like {{c|sys-apps/* -sys-apps/blah}}. ''If this doesn't work, a file can be created called {{f|core-kit-packages-skip}} which contains exclusions, and sys-apps/blah can be added to a line in this file.''  Then, you will want to make sure that sys-apps/blah ''does'' match a package set rule for foobar-kit.
; I want to move a catpkg sys-apps/blah from foobar-kit to core-kit.: To do this, first we'll note that ''foobar-kit comes after core-kit in {{c|kit-groups}}, so core-kit's package set rules will run first. We can thus simply add something that will match 'sys-apps/blah' to core-kit's package-set rules. Once sys-apps/blah is included in core-kit, it will not be available for inclusion in foobar-kit, even if it has an identical rule, or a rule like 'sys-apps/*'. However, note that it is good practice to clean up any rules in foobar-kit that you know are no longer matching any catpkgs.
{{Note|The above two approaches can be used to move catpkgs between kits transparently to the end-user. In the next ego sync, the catpkg will atomically move from one kit to another and no re-emerging will be required, even if the user had emerged the package from the 'old' kit location.}}
; I want to contribute a cool package to Funtoo.: To do this, you will want to open a pull request against [https://code.funtoo.org/bitbucket/projects/CO/repos/flora/browse flora]. Flora is used for all 'bonus' community-contributed ebuilds.
; I want to fix a bug in a particular ebuild.: To do this, first find out where the ebuild is coming from. A good way to do this is to type {{c|ls -d /var/git/meta-repo/kits/*/sys-apps/foobar}}, which will show you what kit it is in. Running {{c|emerge -s sys-apps/foobar}} will also display this information. For research purposes, it is often useful to find where the original catpkg was sourced from. You can consult https://ports.funtoo.org/packages.xml which contains a list of all catpkgs and their source repository. After doing some initial research and seeing what's wrong, you might have a fix for the ebuild. Generally, the best way to fix the ebuild is to fork [https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse kit-fixups] and create an appropriate fixup for the ebuild if none exists, and simply improve our fixup if one exists already. Then you can create a code.funtoo.org pull request, or open a bug on bugs.funtoo.org, or both. Remember that fixup catpkgs will totally replace all upstream ebuilds, so you may need to include multiple versions of the ebuild, even ones that don't need a fix, if they are still needed for certain packages.
{{Note|If you want to fix a bug in an ebuild and you find that the ebuild comes from flora, you will want to fork flora and submit a pull request against flora instead.}}
; I want to make a particular branch of a kit the default kit.: To do this, you will modify {{c|kit_groups}} and set the kit you want to be default to have {{c|'default' : True}} or {{c|'stability' : KitStabilityRating.PRIME}}, or both. Only one kit branch can be set as default.
; I don't want to generate a particular branch of a kit.: To prevent a branch of a kit from being generated, set its {{c|stability}} to {{c|KitStabilityRating.DEPRECATED}} in {{c|kit_groups}}.
; I want to generate a new kit branch that uses much newer ebuilds from Gentoo or from an upstream repo.: First, define a new entry in {{c|kit_source_defs}} that contains the collection of overlays and repos you want to use as sources. Specify the SHA1 commits you want to use for each repo (or don't specify one to use master.) Then, you will want to add a new kit definition to {{c|kit_groups}}, in the "prime" section.
; I want to include a package in Funtoo, but move it to a new name.: To do this, use the "literal with package move" format in the package set (see section on [[#Package Moves]], above). You can also use files in {{f|kit_fixups/move-maps/kitname}} or {{f|kit_fixups/move-maps/global}} (info in a note below the Package Set syntax section.).
{{Important|Be sure to stop by {{c|#funtoo-dev}} on irc.freenode.net if you need further assistance! We are here to help.}}


[[Category:Official Documentation]]
[[Category:Official Documentation]]
[[Category:Development]]
[[Category:Development]]

Latest revision as of 21:47, June 8, 2022

Subpages:
Gitolite Setup
   Important

These pages are out of date. Please see the latest Funtoo:Metatools page for usage information.

Overview

Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.

Before starting, please read and understand the following pages:

Different Approaches

merge-all-kits can be used in a variety of ways. If you are an individual developer, you may want to use merge-all-kits in developer mode, which is its default mode. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.

In developer mode, a meta-repo and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history.

Another way to use merge-all-kits is with gitolite as a remote, which is documented under the Gitolite Setup sub-page. Using this configuration, merge-all-kits will write all of its destination git repositories to gitolite. You can then configure other systems to ego sync from gitolite as a source. This can be good in a distributed development environment. This method can also be used to create a meta-repo and kits that diverge from Funtoo's kit but still connect to its history.

Finally, the merge-all-kits script can be run in a production mode, which is very similar to using it with gitolite except that you may be pushing up your destination meta-repo and kits to a public location like GitHub or GitLab.

When first starting out with merge-all-kits, you should definitely start with developer mode and see if that meets your needs first.

Getting The Code

You can find the code that does this on code.funtoo.org, housed at code.funtoo.org. The script that does all the heavy-lifting is called merge-all-kits. Let's clone it from git, on the machine that will be generating new kits and meta-repo: 

user $ git clone https://code.funtoo.org/bitbucket/scm/core/merge-scripts.git

You will also want to install the following dependencies so that the code can run: 

root # emerge jinja lxml

Configuration and Paths

In general, you should run merge-all-kits as a regular user, and this user needs to be a member of the portage group.

In the user's home directory, create a ~/.merge directory with the following contents:

   /root/.merge 
[sources]

flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
kit-fixups = https://code.funtoo.org/bitbucket/scm/co/flora.git
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git

[branches]

flora = master
kit-fixups = master
meta-repo = master

By default, merge-all-kits will create a ~/repo_tmp directory where it will store all of its work. Inside ~/repo_tmp/source-trees, source git trees that will be used to generate meta-repo will be cloned. Inside ~/repo_tmp/dest-trees, git repositories will be created to write results into. The meta-repo will be stored at /~repo_temp/dest-trees/meta-repo, and in developer mode, the kits will be stored in meta-repo, at ~/repo_tmp/dest-trees/meta-repo/kits, so that meta-repo is configured 'ready to use' at it would appear at /var/git/meta-repo. When 'production' mode is specified via the --prod switch, the meta-repo will not have kits stacked inside it but instead the kits will be written directly into ~/repo_tmp/dest-trees.

A metadata cache will also be created and stored at ~/repo_tmp/metadata-cache, and your user must be a member of the portage group for merge-all-scripts to be able to properly generate the metadata cache.

Running Merge-All-Kits

To run merge-all-kits, use it as follows: 

user $ path/to/merge-scripts/bin/merge-all-kits 1.4-release

The script will then run, and it will git clone flora, kit-fixups and gentoo-staging from the URIs in the ~/.merge file, and assemble them into a complete meta-repo. The meta-repo will be located at $HOME/repo_tmp/dest-trees/meta-repo.

To use the meta-repo for testing, you can add the following lines to /etc/ego.conf:

   /etc/ego.conf 
[global]
meta_repo_path = /home/user/repo_tmp/dest-trees/meta-repo

Then, using ego 2.8.0 or later, run the following command: 

root # ego sync --in-place

This will ensure that the proper kit branches in your meta-repo are checked out and that all the repos.conf and profiles files in /etc/portage are configured correctly for your repository to be used. Note that no actual "git pull" will be performed -- this is intentional and a special feature of the --in-place option -- because your local git repositories have no remotes to pull from and contain all the information they need to be used.

You're now ready to use your meta-repo directly: 

root # emerge foo
Retrieved from "https://www.funtoo.org/index.php?title=Funtoo:Metatools/Advanced_Usage&oldid=40517"