Note:

The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.

Funtoo:Metatools/Advanced Usage/Kit-Fixups Kit Foundations

From Funtoo
< Funtoo:Metatools‎ | Advanced Usage
Revision as of 00:09, January 22, 2020 by Drobbins (talk | contribs) (Created page with "== Changing Repo Definitions == Now that you have generated your first meta-repo, let's look at the key file that defines what overlays and repositories are used to create me...")
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Changing Repo Definitions

Now that you have generated your first meta-repo, let's look at the key file that defines what overlays and repositories are used to create meta-repo, as well as what kit branches exist, and which ones are prime and which are not. You will want to turn your attention to the kit-fixups/modules/fixups/foundations.py file, which can be viewed here on code.funtoo.org]. Let's take a look at various sections of this file:

   foundations.py (python source code) - Top of foundations.py file
#!/usr/bin/python3

from enum import Enum


class KitStabilityRating(Enum):
	PRIME = 0  # Kit is enterprise-quality
	NEAR_PRIME = 1  # Kit is approaching enterprise-quality
	BETA = 2  # Kit is in beta
	ALPHA = 3  # Kit is in alpha
	DEV = 4  # Kit is newly created and in active development
	CURRENT = 10  # Kit follows Gentoo currrent
	DEPRECATED = 11  # Kit is deprecated/retired


def KitRatingString(kit_enum):
	if kit_enum is KitStabilityRating.PRIME:
		return "prime"
	elif kit_enum is KitStabilityRating.NEAR_PRIME:
		return "near-prime"
	elif kit_enum is KitStabilityRating.BETA:
		return "beta"
	elif kit_enum is KitStabilityRating.ALPHA:
		return "alpha"
	elif kit_enum is KitStabilityRating.DEV:
		return "dev"
	elif kit_enum is KitStabilityRating.CURRENT:
		return "current"
	elif kit_enum is KitStabilityRating.DEPRECATED:
		return "deprecated"

At the top of the file, we define an enumeration of the different levels of stability that can exist for a particular kit. Kits marked with a stability rating of KitStabilityRating.PRIME will be tagged in the meta-repo JSON as being a "prime" kit and suitable for production use.

As you scroll down further, you will see some code like this:

   foundations.py (python source code) - foundations.py kit_groups
class KitFoundation:

	kit_groups = {
		'prime': [
			{'name': 'core-kit', 'branch': '1.0-prime', 'source': 'gentoo_prime_protected', 'default': True},
			{'name': 'core-kit', 'branch': '1.1-prime', 'source': 'gentoo_prime_mk3_protected', 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'core-kit', 'branch': '1.2-prime', 'source': 'gentoo_prime_mk4_protected', 'stability': KitStabilityRating.BETA},
			{'name': 'core-hw-kit', 'branch': 'master', 'source': 'funtoo_current', 'default': True},
			{'name': 'security-kit', 'branch': '1.0-prime', 'source': 'gentoo_prime_protected', 'default': True},
			{'name': 'security-kit', 'branch': '1.1-prime', 'source': 'gentoo_prime_mk3_protected', 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'security-kit', 'branch': '1.2-prime', 'source': 'gentoo_prime_mk4_protected', 'stability': KitStabilityRating.BETA},
			{'name': 'xorg-kit', 'branch': '1.17-prime', 'source': 'funtoo_prime_xorg', 'default': False, 'stability': KitStabilityRating.PRIME},
			{'name': 'xorg-kit', 'branch': '1.19-prime', 'source': 'funtoo_mk2_prime', 'default': True, 'stability': KitStabilityRating.PRIME},  # MK2
			{'name': 'gnome-kit', 'branch': '3.20-prime', 'source': 'funtoo_prime_gnome', 'default': True},
			{'name': 'gnome-kit', 'branch': '3.26-prime', 'source': 'funtoo_mk4_prime', 'default': False, 'stability': KitStabilityRating.DEV},
			{'name': 'kde-kit', 'branch': '5.10-prime', 'source': 'funtoo_mk3_prime', 'default': False, 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'kde-kit', 'branch': '5.11-prime', 'source': 'funtoo_prime_kde', 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'kde-kit', 'branch': '5.12-prime', 'source': 'funtoo_prime_kde_late', 'default': True, 'stability': KitStabilityRating.PRIME},
			{'name': 'media-kit', 'branch': '1.0-prime', 'source': 'funtoo_prime_media', 'default': False, 'stability': KitStabilityRating.DEPRECATED},
			{'name': 'media-kit', 'branch': '1.1-prime', 'source': 'funtoo_mk3_prime', 'default': True, 'stability': KitStabilityRating.PRIME},  # MK3
			{'name': 'media-kit', 'branch': '1.2-prime', 'source': 'funtoo_mk4_prime', 'stability': KitStabilityRating.BETA},
			{'name': 'perl-kit', 'branch': '5.24-prime', 'source': 'funtoo_prime_perl', 'default': True},
			{'name': 'perl-kit', 'branch': '5.26-prime', 'source': 'funtoo_mk3_prime', 'default': False, 'stability': KitStabilityRating.DEV},
			{'name': 'python-modules-kit', 'branch': 'master', 'source': 'funtoo_current', 'default': True, 'stability': KitStabilityRating.PRIME},

Here, we see the beginning of the definition of the KitFoundation class, which contains the kit_groups class variable. This variable, as you can see, defines a bunch of kits -- their name, the branch, a stability value that defines the stability rating of the kit, and other values. Note that if a kit is marked as default, it is assumed to have a kit stability rating of PRIME. You will also see that there is a source key, which defines the group of repositories and overlays that are used to create this kit.

Here are three important things to take away from this part of code: First, this is the official master definition of what kits exist, their stability rating, and what overlays/repositories are used to generate each kit. The next thing to take away from this code is that it is certainly possible to modify these settings for your custom meta-repo so that they are different from the Funtoo defaults.

Here is the third and most important thing about kit_groups -- the order that they are defined determines the order which the merge scripts try to match packages. The merge-scripts/package-sets files will be processed in order that the kits are defined in kit_groups, so core-kit first, security-kit second, etc. Once a catpkg has been added to a kit, it will not be added to any successive kits.

Going Deeper into Foundations.py

Moving right along, you will see a section of the KitFoundation class that looks like this:

   foundations.py (python source code) - foundations.py python_kit_settings
python_kit_settings = {
		#	branch / primary python / alternate python / python mask (if any)
		'master': {
			"primary": "python3_6",
			"alternate": "python2_7",
			"mask": None
		},
		'3.4-prime': {
			"primary": "python3_4",
			"alternate": "python2_7",
			"mask": ">=dev-lang/python-3.5"
		},
		'3.6-prime': {
			"primary": "python3_6",
			"alternate": "python2_7",
			"mask": ">=dev-lang/python-3.7"
		},
		'3.6.3-prime': {
			"primary": "python3_6",
			"alternate": "python2_7",
			"mask": ">=dev-lang/python-3.7"
		}
	}

This section is used to define special settings for python-kit. Specifically, for each branch of python-kit, it defines the primary version of python that we will attempt to use to satisfy python-single USE dependencies, followed by an alternate value to use if the ebuild does not support the first. We can also specify a mask to be injected into each python-kit to mask certain versions of python that should be masked due to lack of support in that kit.

Foundations.py Kit Source Definitons

Next, we have a section that looks like this:

   foundations.py (python source code) - foundations.py kit_source_defs
kit_source_defs = {
		"funtoo_current": [
			{"repo": "flora"},
			{"repo": "faustoo"},
			{"repo": "fusion809"},
			{"repo": "gentoo-staging"}
		],
		"funtoo_mk2_prime": [
			{"repo": "flora", },
			{"repo": "faustoo"},
			{"repo": "fusion809", "src_sha1": "489b46557d306e93e6dc58c11e7c1da52abd34b0", 'date': '31 Aug 2017'},
			{"repo": "gentoo-staging", "src_sha1": '80d2f3782e7f351855664919d679e94a95793a06', 'date': '31 Aug 2017'},
			# add current gentoo-staging to catch any new ebuilds that are not yet in our snapshot above (dev-foo/* match)
			{"repo": "gentoo-staging-underlay"},
		],
		"funtoo_mk3_prime": [
			{"repo": "flora", },
			{"repo": "faustoo", },
			{"repo": "fusion809", "src_sha1": "8733034816d3932486cb593db2dfbfbc7577e28b", 'date': '09 Oct 2017'},
			{"repo": "gentoo-staging", "src_sha1": '2de4b388863ab0dbbd291422aa556c9de646f1ff', 'date': '10 Oct 2017'},
			{"repo": "gentoo-staging-underlay"},
		], ...

This section contains definitions of various overlay stacks, called "kit sources". Kit sources are a combination of overlays, arranged in a python list ([ ]). A kit source serves as a unified collection of source catpkgs for a particular kit. Each kit can have one kit source. Kit sources may be shared among kits for consistency purposes, and to avoid duplication and to help organization. Note that this is where we specify branch or SHA1 that is used for each overlay. If no SHA1 is specified, the merge scripts will use the top commit of master as the source for that overlay (ie. the overlay will track upstream.)

   Note

A "catpkg" is a category/package combination, like sys-apps/portage, with no version information. When we match catpkgs, we look for a cat/pkg directory in the overlay, and if we find a match, we grab the full contents of the catpkg directory from the overlay, and ignore any other occurrence of the same catpkg in other overlays. When we talk of "package-set" rules below, we are referring to text files that define what catpkgs go into what kits. We can specify catpkgs literally or use patterns and other approaches to select catpkgs destined for a particular kit. The merge scripts uses these package-set rules files to define what goes in each kit. But where they search for each catpkg is defined in foundations.py, above.

Currently, package-set rules (see note above) are applied in the same order that the overlay appears in the kit_source_defs list -- so for "funtoo_current", package-set rules -- an attempt to match catpkg patterns for a kit -- will be applied to flora first, then faustoo, then fusion809, etc, with gentoo-staging searched last. In theory, this would allow any "upper" overlay to override catpkgs in later-appearing overlays. However, this is not the case, because the "upper" overlays are generally "locked down" so that they are only allowed to provide a particular set of catpkgs, or are allowed to provide any catpkg as long as it doesn't appear in Gentoo, for example. So in actual practice, gentoo-staging is set up to have the higher priority, even though it is searched for matches later. Also notice that newer kit source definitions now have a "bottom" gentoo-staging-underlay definition that has a special purpose -- to allow us to grab new Gentoo catpkgs that aren't in our snapshotted gentoo-staging repository yet. This works because gentoo-staging-underlay tracks upstream master, while gentoo-staging is locked to a particular point in the past. This way we get brand-new gentoo catpkgs, but existing gentoo catpkgs stay locked to a particular point in time.

Completing the Deep Dive

Our deep dive into the functionality of foundations.py is almost complete -- just one more part to go! You will notice the following code at the end of the file:

   foundations.py (python source code) - foundations.py overlay definitions
@property
	def overlays(self):
		return {
			# use gentoo-staging-2017 dirname to avoid conflicts with ports-2012 generation
			"gentoo-staging": {"url": self.config.gentoo_staging, "dirname": "gentoo-staging-2017"},
			"gentoo-staging-underlay": {"url": self.config.gentoo_staging, "dirname": "gentoo-staging-2017-underlay"},
			"faustoo": {"url": "https://github.com/fmoro/faustoo.git", "eclasses": [
				"waf",
				"googlecode"
			],
			            # SKIP any catpkgs that also exist in gentoo-staging (like nvidia-drivers). All others will be copied.
			            "filter": ["gentoo-staging"],
			            # well, I lied. There are some catpkgs that exist in gentoo-staging that we DO want to copy. These are the
			            # ones we will copy. We need to specify each one. This list may change over time as faustoo/gentoo gets stale.
			            "force": [
				            "dev-java/maven-bin",
				            "dev-java/sun-java3d-bin",
				            "dev-php/pecl-mongo",
				            "dev-php/pecl-mongodb",
				            "dev-python/mongoengine",
				            "dev-python/pymongo",
				            "dev-util/idea-community",
				            "dev-util/webstorm",
				            "x11-wm/blackbox"
			            ]
			            },
			"fusion809": {"url": "https://github.com/fusion809/fusion809-overlay.git", "select": [
				"app-editors/atom-bin",
				"app-editors/notepadqq",
				"app-editors/bluefish",
				"app-editors/textadept",
				"app-editors/scite",
				"app-editors/gvim",
				"app-editors/vim",
				"app-editors/vim-core",
				"app-editors/sublime-text"
			]
			              },  # FL-3633, FL-3663, FL-3776
			"plex": {"url": "https://github.com/Ghent/funtoo-plex.git", "select": [
				"media-tv/plex-media-server",
			],
			         },
			# damex's deadbeef (music player like foobar2000) overlay
			"deadbeef": {"url": "https://github.com/damex/deadbeef-overlay.git", "copyfiles": {
				"profiles/package.mask": "profiles/package.mask/deadbeef.mask"
			},
			             },
			# damex's wmfs (window manager from scratch) overlay
			"wmfs": {"url": "https://github.com/damex/wmfs-overlay.git", "copyfiles": {
				"profiles/package.mask": "profiles/package.mask/wmfs.mask"
			},
			         },
			"flora": {"url": self.config.flora, "copyfiles": {
				"licenses/renoise-EULA": "licenses/renoise-EULA"
			},
			          },
		}

This section of foundations.py declares the names for the actual overlays themselves, where we can clone them from, and also helps us answer the question of which overlay has priority over another. In the case of the fusion809 overlay, for example, we will only grab a select set of catpkgs, specified in the select list. Whereas in the faustoo overlay, we will force the merge scripts to grab updates the the catpkgs listed in the force list, but skip any catpkgs in faustoo that also appear in our gentoo-staging snapshot. All these special rules define exceptions to the normal processing logic of the merge scripts, which will look for catpkg matches in the first overlay listed in the kit source definition, before proceeding to the second, etc.

Wrapping Up Foundations.py

We certainly have a lot of levers we can use to control what gets included in a particular kit. Using foundations.py, we can add or remove overlays we want to use, and we can also alter the order they are scanned by the merge scripts. We can control what snapshots are used, and vary this on a per-kit basis. And in the overlay definitions, we can also create exceptions to the normal processing order, to effectively prioritize gentoo catpkgs over overlay catpkgs, even if the overlays are processed first. Or we can limit the scope of what catpkgs we scan a particular overlay for, or force certain catpkgs to be copied from a particular overlay if we know we want those particular ones.

All that is left to cover are the package-set rules, which define which catpkgs go into each kit, and kit-fixups, which allow us to have Funtoo overrides for any catpkgs we want. We'll cover those topics next.

Retrieved from "https://www.funtoo.org/index.php?title=Funtoo:Metatools/Advanced_Usage/Kit-Fixups_Kit_Foundations&oldid=30818"