Metatools 1.1.0

Metatools 1.1.0 is a major release which was released on 19 August 2022.

ChangeLog

The 1.1.0 release of metatools has a number of significant updates:

• Major bug fixes in a lot of areas, including python-3.7 compatibility fixes.
• Massive improvements to error handling and ensuring you get a useful traceback when there is a problem.
• Major new features related to YAML autogens.
• Support for "closed loop fastpull" (better distfile handling for CDN).
• Useful new command-line arguments for tools.
• Major improvements to the github-1 generator and associated github.py metatools support code.
• Dynamic archives support (powerful new feature)

   Important

Please note that metatools requires a version earlier than 0.23.0 of httpx. 0.23.0 introduced breaking changes to the httpx API. Please see FL-9887 and FL-9888.

Contributors

Thanks to the following contributors who helped to make this release better:

• User:Invakid404
• User:Siris
• User:Alex2101

Dynamic Archives Support

As described in FL-9270, it is now possible for an autogen to actually create its own tarballs which magically get added to the Funtoo CDN infrastructure with no additional steps on the autogen developer's part. Full documentation for this feature as well as a FAQ can be found here in the metatools repo: https://code.funtoo.org/bitbucket/users/drobbins/repos/funtoo-metatools/browse/docs/features/dynamic-archives.rst

New Command-Line Arguments

There are several useful, new command-line arguments.

Command-Line Arguments -- doit

• You can now use --pkg to specify the package name from the YAML you want to run autogen for. No other autogens will be run.
• You can now use --cat to specify the category from the YAML you want to run autogen for. This can be combined with --pkg.
• You can now specify the specific autogen.yaml and autogen.py files you want to auto-generate as positional arguments on the command-line. If no files are specified, the default behavior of recursively generating all autogens from the current path will occur. Otherwise, only those autogens specified will be executed. This can also be combined with the --pkg and --cat options.

Autogen YAML Improvements

There is significant new functionality related the YAML autogen syntax used in autogen.yaml in a number of areas:

YAML - Multi-Arch Generators

As described in FL-9443, it is now easily possible to support GitHub projects that have arch-specific assets in their releases. This is demonstrated in the prometheus-bin autogen and works as follows. In an autogen.yaml you will have something like this for assets:

prometheus:
  generator: github-1
  packages:
    - prometheus-bin:
        assets:
          amd64: prometheus-{version}.linux-amd64.tar.gz
          arm64: prometheus-{version}.linux-arm64.tar.gz
        template: prometheus-bin.tmpl

In the template, you will have something like this for SRC_URI:

SRC_URI="
        {%- for arch, artifact in artifacts.items() %}
        {{ arch }}? ( {{ artifact.src_uri }} )
        {%- endfor %}"

YAML - Merged Defaults

Implemented FL-9664 to allow intelligent merging of defaults sections. Prior to this, subsections of YAML were not intelligently merged. For example, in the following example, the github/user would not get properly set, as the entire specific github section in the YAML for the foobar package would override the default value:

my_autogen:
  defaults:
    github:
      user: foobaroni
  packages:
    - foobar:
        github:
          repo: foobar_is_awesome

With the new implementation, any lists that exist both in defaults and in the specific package area will be appended to, and any dictionary/objects will have additional augmented attributes rather than overwriting the entire section.

This allows YAML to be cleanly extended and behaves intuitively. Defaults can be overridden but will generally not be discarded. This behavior also works with the "invisible defaults" that exist when specifying multiple package versions:

my_autogen:
  packages:
    - foobar:
        github:
          user: foobaroni
        version:
          1.0.0:
            github:
              repo: foobar_is_awesome

Above, github/user under packages/foobar sets defaults for all versions of packages under /packages/foobar/version, and previously, github/user would be overwritten whereas now both user and repo will be specified as arguments for the 1.0.0 version of foobar.

YAML - Multiple Versions

As described in FL-9856, it is now possible to specify multiple versions in autogen.yaml for any package. To do this, instead of specifying a "version: <version>" string to hard-code a version, or leaving it blank to imply "latest version", you can make version: an element with multiple version sections, each with their own version-specific settings if needed. For example, here is the kitty autogen:

kitty_rule:
  generator: github-1
  defaults:
    tarball: kitty-{version}.tar.xz
    github:
      user: kovidgoyal
      repo: kitty
      query: releases
    revision:
      0.25.0: 1
    version:
      0.25.1:
        python_compat: python3+

If you also want the latest version autogenned, use the special keyword "latest" for the version, which will result in the version being blank in pkginfo (indicating latest should be generated.) Metatools will turn each version specified into its own distinct autogen generate() call.

New GitHub Features

The work on our GitHub API has been very significant and deserves its own documentation. This will be part of a future metatools release which will cover the metatools GitHub API in detail.