Difference between revisions of "Funtoo:Metatools/Advanced Usage"

Latest revision as of 21:47, June 8, 2022

Subpages:

Important

These pages are out of date. Please see the latest Funtoo:Metatools page for usage information.

Overview

Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.

Before starting, please read and understand the following pages:

Different Approaches

merge-all-kits can be used in a variety of ways. If you are an individual developer, you may want to use merge-all-kits in developer mode, which is its default mode. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.

In developer mode, a meta-repo and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history.

Another way to use merge-all-kits is with gitolite as a remote, which is documented under the Gitolite Setup sub-page. Using this configuration, merge-all-kits will write all of its destination git repositories to gitolite. You can then configure other systems to ego sync from gitolite as a source. This can be good in a distributed development environment. This method can also be used to create a meta-repo and kits that diverge from Funtoo's kit but still connect to its history.

Finally, the merge-all-kits script can be run in a production mode, which is very similar to using it with gitolite except that you may be pushing up your destination meta-repo and kits to a public location like GitHub or GitLab.

When first starting out with merge-all-kits, you should definitely start with developer mode and see if that meets your needs first.

Getting The Code

You can find the code that does this on code.funtoo.org, housed at code.funtoo.org. The script that does all the heavy-lifting is called merge-all-kits. Let's clone it from git, on the machine that will be generating new kits and meta-repo:

user $ git clone https://code.funtoo.org/bitbucket/scm/core/merge-scripts.git

You will also want to install the following dependencies so that the code can run:

root # emerge jinja lxml

Configuration and Paths

In general, you should run merge-all-kits as a regular user, and this user needs to be a member of the portage group.

In the user's home directory, create a ~/.merge directory with the following contents:

/root/.merge

[sources]

flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
kit-fixups = https://code.funtoo.org/bitbucket/scm/co/flora.git
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git

[branches]

flora = master
kit-fixups = master
meta-repo = master

By default, merge-all-kits will create a ~/repo_tmp directory where it will store all of its work. Inside ~/repo_tmp/source-trees, source git trees that will be used to generate meta-repo will be cloned. Inside ~/repo_tmp/dest-trees, git repositories will be created to write results into. The meta-repo will be stored at /~repo_temp/dest-trees/meta-repo, and in developer mode, the kits will be stored in meta-repo, at ~/repo_tmp/dest-trees/meta-repo/kits, so that meta-repo is configured 'ready to use' at it would appear at /var/git/meta-repo. When 'production' mode is specified via the --prod switch, the meta-repo will not have kits stacked inside it but instead the kits will be written directly into ~/repo_tmp/dest-trees.

A metadata cache will also be created and stored at ~/repo_tmp/metadata-cache, and your user must be a member of the portage group for merge-all-scripts to be able to properly generate the metadata cache.

Running Merge-All-Kits

To run merge-all-kits, use it as follows:

user $ path/to/merge-scripts/bin/merge-all-kits 1.4-release

The script will then run, and it will git clone flora, kit-fixups and gentoo-staging from the URIs in the ~/.merge file, and assemble them into a complete meta-repo. The meta-repo will be located at $HOME/repo_tmp/dest-trees/meta-repo.

To use the meta-repo for testing, you can add the following lines to /etc/ego.conf:

/etc/ego.conf

[global]
meta_repo_path = /home/user/repo_tmp/dest-trees/meta-repo

Then, using ego 2.8.0 or later, run the following command:

root # ego sync --in-place

This will ensure that the proper kit branches in your meta-repo are checked out and that all the repos.conf and profiles files in /etc/portage are configured correctly for your repository to be used. Note that no actual "git pull" will be performed -- this is intentional and a special feature of the --in-place option -- because your local git repositories have no remotes to pull from and contain all the information they need to be used.

You're now ready to use your meta-repo directly:

root # emerge foo

@@ Line 1: / Line 1: @@
-{{Subpages|Gitolite Setup,Kit-Fixups Kit Foundations,Package Sets and Move Maps}}
+{{Subpages|Gitolite Setup}}
+{{Important|These pages are out of date. Please see the latest [[Funtoo:Metatools]] page for usage information.}}
 == Overview ==
@@ Line 5: / Line 7: @@
 Funtoo Linux uses "merge scripts" to create its kits and update meta-repo. These scripts work by sourcing ebuilds from various overlays, and combining them using special algorithms to yield the kits you use. A meta-repo is also generated, which points to the specific kits generated that are designed to work together.
-Required for merge scripts is  {{c|dev-python/lxml}}.
+Before starting, please read and understand the following pages:
+* [[Kit-fixups]]
+* [[Kit-fixups/FAQ]]
 == Different Approaches ==
-{{c|merge-all-kits}} can be used in a variety of ways. If you are an individual developer, you may want to use {{c|merge-all-kits}} in ''developer mode'', which is its default configuration. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.
+{{c|merge-all-kits}} can be used in a variety of ways. If you are an individual developer, you may want to use {{c|merge-all-kits}} in ''developer mode'', which is its default mode. Developer mode makes it easy to generate a meta-repo and kits that you can test locally.
-In developer mode, a {{c|meta-repo}} and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk locally, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history. When
+In developer mode, a {{c|meta-repo}} and all automatically-generated kit git repositories are created from scratch. These git repositories will have no remotes, so they will simply exist locally on disk, and they will not have any history shared with the Funtoo project. It you look at the commit history for the kits and meta-repo, you will see that they essentially have no history.
 Another way to use {{c|merge-all-kits}} is with {{c|gitolite}} as a remote, which is documented under the [[#Gitolite Setup|Gitolite Setup]] sub-page. Using this configuration, {{c|merge-all-kits}} will write all of its destination git repositories to gitolite. You can then configure other systems to {{c|ego sync}} from gitolite as a source. This can be good in a distributed development environment. This method can also be used to create a meta-repo and kits that ''diverge'' from Funtoo's kit but still connect to its history.
@@ Line 21: / Line 25: @@
 == Getting The Code ==
-You can find the code that does this on code.funtoo.org, housed at https://code.funtoo.org/bitbucket/projects/CORE/repos/merge-scripts/browse. The script that does all the heavy-lifting is called {{c|merge-all-kits}}. Let's clone it from git, on the machine that will be generating new kits and meta-repo:
+You can find the code that does this on code.funtoo.org, housed at [https://code.funtoo.org/bitbucket/projects/CORE/repos/merge-scripts/browse code.funtoo.org]. The script that does all the heavy-lifting is called {{c|merge-all-kits}}. Let's clone it from git, on the machine that will be generating new kits and meta-repo:
 {{console|body=
@@ Line 33: / Line 37: @@
 }}
-== Configuration ==
+== Configuration and Paths ==
 In general, you should run {{c|merge-all-kits}} as a regular user, and this user needs to be a member of the {{c|portage}} group.
@@ Line 52: / Line 56: @@
 meta-repo = master
 }}
+By default, {{c|merge-all-kits}} will create a {{c|~/repo_tmp}} directory where it will store all of its work. Inside {{f|~/repo_tmp/source-trees}}, source git trees that will be used to generate meta-repo will be cloned. Inside {{f|~/repo_tmp/dest-trees}}, git repositories will be created to write results into. The meta-repo will be stored at {{c|/~repo_temp/dest-trees/meta-repo}}, and in developer mode, the kits will be stored in meta-repo, at {{c|~/repo_tmp/dest-trees/meta-repo/kits}}, so that meta-repo is configured 'ready to use' at it would appear at {{f|/var/git/meta-repo}}. When 'production' mode is specified via the {{c|--prod}} switch, the meta-repo will not have kits stacked inside it but instead the kits will be written directly into {{c|~/repo_tmp/dest-trees}}.
+A metadata cache will also be created and stored at {{c|~/repo_tmp/metadata-cache}}, and your user must be a member of the {{c|portage}} group for {{c|merge-all-scripts}} to be able to properly generate the metadata cache.
 == Running Merge-All-Kits ==
@@ Line 83: / Line 91: @@
 # ##i##emerge foo
 }}
-== Kit Fixups ==
-And finally, we have saved a very key part of the kit generation process for last. The {{c|kit-fixups}} repository is so named because it contains ''fixups'', which are forked Funtoo catpkgs that are used to override catpkgs that appear in the upstream overlays and repositories. They have a special structure. We will look at the structure of the core-kit fixup directory, although others will follow the same model:
-{{TableStart}}
-<tr><th>CatPkg Path</th><th>Description</th></tr>
-<tr><td>{{f|kit-fixups/core-kit/global/sys-apps/portage}}</td><td>Due to the {{c|global}} directory, this catpkg will always be used when a package set specifies a match for {{c|sys-apps/portage}}, for '''all''' branches of core-kit.</td></tr>
-<tr><td>{{f|kit-fixups/core-kit/curated/sys-apps/portage}}</td><td>Due to the {{c|curated}} directory, this catpkg will always be used when a package set specifies a match for {{c|sys-apps/portage}}, for '''all''' branches of core-kit '''except a master branch.'''</td></tr>
-<tr><td>{{f|kit-fixups/core-kit/1.2-prime/sys-apps/portage}}</td><td>Due to the {{c|1.2-prime}} directory, this catpkg will always be used when a package set specifies a match for {{c|sys-apps/portage}}, for the '''1.2-prime''' branch of core-kit only.</td></tr>
-{{TableEnd}}
-Remember that kit-fixups is designed so that ''a fixup will always override any upstream packages.'' This makes it easy to keep track of Funtoo-maintained core packages. And also note that the flora repository should be used for "bonus" packages while kit-fixups should focus more on forks of critical system packages and bug fixes for Funtoo. This way, we can keep contributed ebuilds separate from core operating system ebuilds and associated bug fixes for upstream issues.
-== Developer Q&A ==
-This section contains various tasks that a developer may need to perform, and what steps should be taken to perform each of these steps.
-; I want to move a catpkg sys-apps/blah from core-kit to foobar-kit.: To do this, first we'll note that ''core-kit comes before foobar-kit in {{c|kit_groups}}. This means that core-kit's package set rules will run first. So we will want to make sure that sys-apps/blah does '''not''' match any rules in the core-kit package-set. This can be done by possibly removing a package-set rule, or using a wildcard with exclusion like {{c|sys-apps/* -sys-apps/blah}}. ''If this doesn't work, a file can be created called {{f|core-kit-packages-skip}} which contains exclusions, and sys-apps/blah can be added to a line in this file.''  Then, you will want to make sure that sys-apps/blah ''does'' match a package set rule for foobar-kit.
-; I want to move a catpkg sys-apps/blah from foobar-kit to core-kit.: To do this, first we'll note that ''foobar-kit comes after core-kit in {{c|kit-groups}}, so core-kit's package set rules will run first. We can thus simply add something that will match 'sys-apps/blah' to core-kit's package-set rules. Once sys-apps/blah is included in core-kit, it will not be available for inclusion in foobar-kit, even if it has an identical rule, or a rule like 'sys-apps/*'. However, note that it is good practice to clean up any rules in foobar-kit that you know are no longer matching any catpkgs.
-{{Note|The above two approaches can be used to move catpkgs between kits transparently to the end-user. In the next ego sync, the catpkg will atomically move from one kit to another and no re-emerging will be required, even if the user had emerged the package from the 'old' kit location.}}
-; I want to contribute a cool package to Funtoo.: To do this, you will want to open a pull request against [https://code.funtoo.org/bitbucket/projects/CO/repos/flora/browse flora]. Flora is used for all 'bonus' community-contributed ebuilds.
-; I want to fix a bug in a particular ebuild.: To do this, first find out where the ebuild is coming from. A good way to do this is to type {{c|ls -d /var/git/meta-repo/kits/*/sys-apps/foobar}}, which will show you what kit it is in. Running {{c|emerge -s sys-apps/foobar}} will also display this information. For research purposes, it is often useful to find where the original catpkg was sourced from. You can consult https://ports.funtoo.org/packages.xml which contains a list of all catpkgs and their source repository. After doing some initial research and seeing what's wrong, you might have a fix for the ebuild. Generally, the best way to fix the ebuild is to fork [https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse kit-fixups] and create an appropriate fixup for the ebuild if none exists, and simply improve our fixup if one exists already. Then you can create a code.funtoo.org pull request, or open a bug on bugs.funtoo.org, or both. Remember that fixup catpkgs will totally replace all upstream ebuilds, so you may need to include multiple versions of the ebuild, even ones that don't need a fix, if they are still needed for certain packages.
-{{Note|If you want to fix a bug in an ebuild and you find that the ebuild comes from flora, you will want to fork flora and submit a pull request against flora instead.}}
-; I want to make a particular branch of a kit the default kit.: To do this, you will modify {{c|kit_groups}} and set the kit you want to be default to have {{c|'default' : True}} or {{c|'stability' : KitStabilityRating.PRIME}}, or both. Only one kit branch can be set as default.
-; I don't want to generate a particular branch of a kit.: To prevent a branch of a kit from being generated, set its {{c|stability}} to {{c|KitStabilityRating.DEPRECATED}} in {{c|kit_groups}}.
-; I want to generate a new kit branch that uses much newer ebuilds from Gentoo or from an upstream repo.: First, define a new entry in {{c|kit_source_defs}} that contains the collection of overlays and repos you want to use as sources. Specify the SHA1 commits you want to use for each repo (or don't specify one to use master.) Then, you will want to add a new kit definition to {{c|kit_groups}}, in the "prime" section.
-; I want to include a package in Funtoo, but move it to a new name.: To do this, use the "literal with package move" format in the package set (see section on [[#Package Moves]], above). You can also use files in {{f|kit_fixups/move-maps/kitname}} or {{f|kit_fixups/move-maps/global}} (info in a note below the Package Set syntax section.).
 [[Category:Official Documentation]]
 [[Category:Development]]