Metatools 1.3.0

Metatools 1.3.0 is a major release which was released on 29 May 2023.

ChangeLog

This is a feature release containing a number of new capabilities and improvements.

• Refactor of how we handle the --immediate option internally to be more intuitive in the source code.
• Initial implementation of ZeroMQ-based "moonbeam" communications framework to allow child doit processes to communicate with the master merge-kits process. This will initially be used to implement logging of all issues encountered during the merge-kits run so we can generate a nice summary of problems (see FL-11179). The initial framework has just been added but the logging/reporting functionality is not yet implemented.
• When running ensure_fetched(), use an asyncio.wait(0) to allow scheduling/execution of new asyncio tasks. This method often gets hammered with hundreds of new requests and this can stall out existing async tasks (like when a bunch of crates or go modules are getting queued for download all at once.)
• In doit and merge-kits, a large conversion from more thread-oriented to single-process async (with forked subprocesses for external commands) whenever possible, keeping threads only for CPU parallelization for Portage metadata generation. This allows "moonbeam" to be able to send/receive messages efficiently when other stuff is going on.
• Python 3.7 compatibility restored to the codebase (I made a minor change which made the code 3.9+.)
• Add --howdy argument for merge-kits which causes "HOWDY" to be printed every 0.1 seconds from the moonbeam ZeroMQ engine. This is used to test for any issues related to async tasks not being scheduled to run frequently. If you don't see HOWDY printed continuously then some long-running task is blocking the async event loop and this should probably be fixed.
• Fix a 3-year-old bug where the Python USE-optimization code was not generating deterministic results in package.use files.
• Misc fixes to pyhelper to introduce sorting in some areas to reduce randomization (non-deterministic order) of elements in ebuilds.
• Reimplementation of deepdive. Add an advanced deepquery that can actually rewrite packages.yaml files for us to remove unused ebuilds automatically. This is an active area of work and needs some docs and cleanup.
• When specifying assets: for github-1 to grab, add a special keyword "<source.tar.gz>" literal string which allows you to grab the auto-generated default tarball. There was not previously a way to grab this as well as other assets that were uploaded to a release.
• Support ETag and Last-Modified HTTP 304 responses. This dramatically improves API limits for GitHub, etc.
• FL-11369: tweak to rust.py to background and make the cargo update async-compatible.
• Deprecate max_age= parameter for fetching (this was a vestigial thing that was not being used.)
• As part of the work on HTTP 304 support, spider.http_fetch now returns a tuple of headers and content. This is necessary so we can extract "Last-Modified" and "ETag" headers and store them in the fetch cache so we can use them for successive requests for the HTTP 304 support.
• Fix an issue with doit that is common to all Python programs -- All python programs will attempt to import things from the current working directory if some directory exists with the same name as a module it needs. This is really, really dumb. This caused doit to fail in python-modules-kit, inside curated/dev-python, due to the "click" directory existing after first doit is run, which then caused successive doits to fail when httpx tries to import the click module.
• Cleaned up some error output issues.
• FL-11300: attempt to address Tree OOP hierarchy to ensure initialize is available for AutoCreatedGitTree class. (Thanks: borisp)