Funtoo:Metatools

   Summary

This project is focused on the development of Funtoo's metatools and autogeneration technology.

View Scope...

   Subpages

• Releases

   People

Leads

• drobbins

Contributors

• alex2101
• invakid404

   Latest Status

Metatools 1.1.0 has been released, which includes a lot of new features and improvements.

19 August 2022

Metatools (git) (pypi) is the framework used by Funtoo Linux to create meta-repo and autogenerate ebuilds. It is a foundational tool for Funtoo Linux development. It is also a living experiment exploring how we can develop and maintain ebuilds more efficiently and effectively.

The latest release of metatools is 1.3.5, released on 09 November 2023. It is a regular software release. View Release Notes.

What is an "Autogen"?

Quite simply, an "autogen" is kind of like an ebuild generator, using the metatools Python framework to dynamically generate ebuilds. A key functionality of an autogen is to allow ebuilds to be generated to reflect the current version of source code that is available from upstream.

Where can I Find Autogens?

You can find autogens by looking in the kit-fixups repository (git) for any files named:

1. autogen.py
2. autogen.yaml

Both YAML and Python autogens use our metatools framework -- the YAML variants are higher-level autogens that feed metatools "generators", kind of like a pipeline. As of Jan 29, 2022, all Funtoo Linux source overlays -- kit-fixups as well as others -- support the creation of autogens. What this means is that it's possible to write an autogen for anything in Funtoo to ensure that it is always kept up-to-date.

Installation

To install metatools, emerge it:

root # emerge metatools

Autogen Setup

The command in metatools that performs autogeneration locally is called doit. Prior to using this command locally, it's recommended to create an ~/.autogen file under your user account that contains a GitHub personal access token. This will allow doit to use it when querying GitHub and will prevent doit from hitting API limits:

   ~/.autogen

authentication:
  api.github.com:
    username: danielrobbins
    password: <insert access token string here>

For more information on setting up GitHub personal access tokens, see: https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token

Autogen Usage

Funtoo's meta-repo is defined in the kit-fixups repository (git), which is the starting point for all autogeneration of ebuilds.

To perform your first doit run, enter into the appropriate kit-fixups directory and run the doit command:

user $ cd development/kit-fixups/core-kit/curated
user $ doit
[INFO 21:02:35] Fetched https://git-scm.com/download/linux 6042 bytes
[INFO 21:02:35] Created: dev-vcs/git/git-2.35.1.ebuild
[INFO 21:02:35] Fetched https://api.github.com/repos/hashicorp/vagrant_cloud/tags 9513 bytes
[INFO 21:02:36] Created: dev-ruby/vagrant_cloud/vagrant_cloud-2.0.3.ebuild
[INFO 21:02:36] Fetched https://api.github.com/repos/distcc/distcc/releases 34549 bytes
[INFO 21:02:36] Created: sys-devel/distcc/distcc-3.4.ebuild
[INFO 21:02:36] Fetched https://api.github.com/repos/dbrgn/tealdeer/releases 151336 bytes
[INFO 21:02:36] Fetched https://api.github.com/repos/tmux/tmux/releases 94211 bytes
[INFO 21:02:36] Created: app-misc/tmux/tmux-3.2a.ebuild
[INFO 21:02:36] Fetched https://api.github.com/repos/lotabout/skim/releases 225892 bytes
[INFO 21:02:36] Fetched https://ftp.debian.org/debian/pool/main/c/ca-certificates/ 7174 bytes
[INFO 21:02:36] Created: app-misc/ca-certificates/ca-certificates-20211016.ebuild
[INFO 21:02:36] Fetched https://api.github.com/repos/jarun/nnn/releases 564842 bytes
[INFO 21:02:36] Fetched https://api.github.com/repos/netwide-assembler/nasm/tags 13895 bytes
[INFO 21:02:36] Created: app-misc/nnn/nnn-4.4.ebuild
[INFO 21:02:36] Fetched https://api.github.com/repos/doxygen/doxygen/tags?per_page=100 46121 bytes

What you are seeing above is output from the doit command, which will:

1. Find all autogens (autogen.yaml and autogen.py files) in the current directory tree.
2. Execute them using multiple threads, during which doit will:
   1. Query upstream Web sites and APIs (like GitHub) for current versions of source code,
   2. Fetch all source code artifacts (distfiles) by using an efficient spider engine, and
   3. Auto-create ebuilds for these packages, along with Manifests.

Funtoo contributors will use the doit command locally to test their autogens and ensure their proper operation before submitting a PR.

Where Can I Learn More about Metatools?

The best place to learn is to join us in the #metatools and #dev-help channels on Discord.

There is "in progress" documentation for metatools at our to-be-launched projects site: https://projects.funtoo.org/metatools/docs/install.html SSL Cert is not yet valid and the documentation is a work in progress. Refer to this wiki page for current setup documentation until the documentation has caught up with the current release of metatools.

Status Updates

2022-08-19

Metatools 1.1.0 has been released, which includes a lot of new features and improvements.

2022-04-15

Metatools 1.0.3 has been released, with a bunch of improvements including support for running merge-kits.

2022-03-15

Metatools git master now can successfully regen the tree in "dev" mode. It doesn't use mongodb, or threading like the previous incarnation, but is still faster than the older threaded version. This is an important milestone -- soon we will be using this version for production tree regens.

2022-02-28

Drobbins is working on "master" branch, specifically merge-kits, and getting this working fully (currently, "legacy" branch is used for tree regen). This involves upgrading release YAML, implementing support for "sourced" kits, and testing.

2022-02-22

drobbins is working on the "master" branch of metatools. With the 1.0.2 release, mongodb is no longer a dependency.