Difference between revisions of "Funtoo:Harvester"

Revision as of 20:27, September 25, 2022

Latest Status

harvester/2023-11 has been merged into official Funtoo, and harvester/2024-02 has been created. Please hold off on PRs to harvester. We are going to define a more structured approach to harvester development very soon so we can move forward strategic initiatives.

01 February 2024

Important

To submit a PR for harvester, please send a PR via code.funtoo.org and target the harvester/2022-09 branch of kit-fixups. coffnix is the master of the harvest and you should be sure to stay in contact with him in the #harvester channel on Funtoo Discord if you are using harvester.

Welcome to the harvester project!

The harvester project is focused on providing a meta-repo that can be used for active development by Funtoo Linux contributors. This allows evaluation of experimental and potentially disruptive changes in a completely separate tree without impacting regular Funtoo Linux installs.

The Harvester Concept

Harvester exists as a branch of kit-fixups, which is listed at the top of this page. harvester.funtoo.org generates a tree using this special branch, containing experimental changes. These changes are made available via git from harvester.funtoo.org directly.

Frequently Asked Questions

What issues are you solving with harvester?: We need a "testing release" to test imminent changes to our releases, compatible with CI (continuous integration), to accelerate development and improve quality and test-ability of our imminent changes before they hit production systems.

How does this relate to next-release in general? I thought that this was supposed to be the "experimental" branch?: next-release was meant to be a very up-to-date rolling release but still actually be useful for real work. So, next-release should not become a mechanism to integration-test potentially disruptive changes to Funtoo, nor is it a good candidate for CI to run tests. This has become more important now that next-release is the official recommended default install for new Funtoo Linux systems, due to it being much more current than 1.4-release. This means that we should keep it more current, but also keep it functioning well and improve its reliability and usefulness. Thus, we have a harvester next-release where we can test and stabilize things before they are rolled into the official next-release, allowing problems to be identified proactively -- before they impact production systems. This will actually speed up development of next-release.

Why not create a unstable-release, then? Why harvester?: Good question. Creating a new unstable-release is not a useful option because it will diverge from next-release over time and then ultimately we will re-create the issue we are trying to solve, which is not having a CI-compatible release where we can push potentially breaking changes for manual and automated testing to something that very closely resembles our current production releases. Harvester also can provide us with a CI-friendly fork of all of our releases, so it can allow us to test potentially breaking changes destined for any of our releases.

What releases are available in harvester?: The next release is currently being generated and is available from harvester.funtoo.org (See How To Use, below.)

Is there a 1.4-release harvester tree?: We are not currently generating a harvester version of 1.4-release, but this could be added in the future.

How will harvester be synchronized with the official releases?: We have options here, but the overall concept is that we have a kit-fixups branch like harvester/2022-09 that is always forked from the official upstream releases of Funtoo. So periodically, we re-sync with the official release by creating a new timestamped harvester branch, before adding experimental changes. The frequency is something we control.

How will changes in harvester be added to official releases?: Again, we have options here. If we have a bunch of experimental changes in harvester, and they all get worked on and become quite reliable, we can directly merge the harvester branch into master. Or, if only some changes become stable, the ones that are stable can be sent as PR's into master (the official releases.) Or, if we try a bunch of things that are a total disaster, we can abandon the timestamped harvester branch and create a new one, and just abandon the changes. So we have the ability to adapt as needed and don't need to have a 'set in stone' git flow.

How does this work with gnome-kit and other sourced kits which have their own git repositories, such as gnome-kit-sources?: We are working on a script that will create harvester branches for all sourced kits, so very soon you will have a harvester/2022-09/3.36-prime branch in gnome-kit-sources which can receive experimental GNOME changes. For now, we can create these branches manually as needed.

Developer FAQ

How should I coordinate with the harvester community?: Great question! You must be active in the #harvester Discord channel to talk with us there. Harvester is run by drobbins (BDFL),Adbosco (Inspector General) and Coffnix (Master of the Harvest). We each do different things. The BDFL oversees and directs the effort. The Inspector General is actively looking at the quality of various PRs and overseeing the integration testing efforts, and the Master of the Harvest is operating and maintaining the Harvester CI infrastructure.

What PRs should I send to harvester instead of directly to next-release?: Anything that is potentially highly disruptive and could break other packages should be sent to harvester first. This will allow your changes to become part of the harvester next-release so you can see the impact of your changes on systems that are set up to use harvester (See How To Use, below.) If your PR fixes something that is already broken (break/fix) and is very unlikely to make things worse, or adds new functionality that doesn't impact other ebuilds, it is generally still safe to send this PR directly to the official releases of Funtoo.

I still don't understand what I should send to harvester or the official release.: OK. Then pick one. If you send your PR to the official release and we determine that it is potentially disruptive or could break other ebuilds, we will identify this, decline the PR, and ask you to re-send it to harvester. If you send it to harvester, we will likely approve it so you can see the results of your changes when harvester is regenned. You can then continue integration-testing your changes in harvester until they are ready to be released. At this point, please send a PR for the official release and document the testing you performed in the PR description.

How do I "integration-test" my harvester changes?: See How To Use, below. You will set up a Funtoo system (or container, using LXD) and set it up to use harvester. You will then sync the harvester tree once your changes are "live" in harvester, and you can see how your system behaves with these changes. By doing this, you can see the impact of your changes on Funtoo as a whole, and identify potential negative impact such as dependencies that do not resolve or build problems when your changes are active. This provides better testing that just manually using ebuild to test-build your changes locally. It also allows others to test your changes on their harvester systems. Be sure to leverage the #harvester Discord community to request testing of your changes as needed.

Should I send a PR simultaneously for both harvester and the official release?: In general, no. If it's ready for the official release, send it there. We will periodically merge ongoing changes from the official release into harvester so they stay in-sync as possible.

Do I need to worry about "blowing up" harvester with a really bad PR?: Most of the time, no, you don't need to be extremely concerned about this, although we ask that your PRs have undergone as reasonable local testing as possible and you don't send any "crap shot" PR's to harvester. If your PR contains a broken autogen which prevents the tree from regenning, it may be reverted and we will reach out to you in the #harvester Discord channel to let you know.

How To Use

Currently, harvester only provides an experimental next-release tree.

To use harvester on an existing next-release system, add the following lines to your /etc/ego.conf:

/etc/ego.conf

[global]
release = next
sync_base_url = git://harvester.funtoo.org/funtoo/tree/{repo}

Then, back up your existing meta-repo, and re-sync the new harvester meta-repo:

root # cd /var/git
root # mv meta-repo meta-repo.bak
root # ego sync

Your system will now have the harvester next-release meta-repo available to it, so you can test the latest experimental changes in harvester via emerge.

How To Submit

As with all PR's, it is required that you open an issue on bugs.funtoo.org. Once this is done, and the issue has been triaged so that is no longer in the "Intake" state, it is possible to start progress on the issue and submit a PR referencing the issue, targeting the kit-fixups branch listed at the top of the page. The PR will be on an accelerated track to be merged into harvester. When the harvester tree is next regenerated, the change will appear in the tree and ego sync will make it available on any Funtoo Linux systems using harvester.

Once your changes have been sufficiently tested, it is then possible to submit a vetted PR to the production next-release tree, by submitting a PR to the kit-fixups master branch.

Using Sourced Kits

Most kits can receive new ebuilds and autogens via kit-fixups, and thus it is easy to add experimental ebuilds and autogens to most kits, but there are exceptions.

Any "sourced" kits are kits that are fully self-contained in their own repos, such as gnome-kit-sources. You can find a full list of sourced kits by looking at kit-fixups/browse/releases/next/repositories.yaml -- any kit with kind: sourced is a sourced kit. This means that kit-fixups isn't used to "tweak" the contents of the kit -- instead, the ebuilds and autogens come from the referenced branch as-is, and all updates appear in this separate "sources" repo and branch.

This creates a problem for harvester -- how do you inject experimental changes into sourced kits? The answer is to coordinate with our master of the harvest so that we can create a special harvester branch on the sourced kit for you, and update our YAML so that this harvester branch is used for our harvester tree. Basically, we will create a harvester/YYYY-MM branch in the sourced kit, and then you submit the PR against that. We don't do this automatically since many kits are not sourced and the creation of these branches is not yet automated.

Note

We will probably automate the creation of harvester branches for sourced kits in the near future, and add details here about how to submit to harvester for sourced kits.

@@ Line 44: / Line 44: @@
 ;I still don't understand what I should send to harvester or the official release.: OK. Then pick one. If you send your PR to the official release and we determine that it is potentially disruptive or could break other ebuilds, we will identify this, decline the PR, and ask you to re-send it to harvester. If you send it to harvester, we will likely approve it so you can see the results of your changes when harvester is regenned. You can then continue integration-testing your changes in harvester until they are ready to be released. At this point, please send a PR for the official release and document the testing you performed in the PR description.
+;How do I "integration-test" my harvester changes?: See [[#How To Use|How To Use]], below. You will set up a Funtoo system (or container, using [[LXD]]) and set it up to use harvester. You will then sync the harvester tree once your changes are "live" in harvester, and you can see how your system behaves with these changes. By doing this, you can see the impact of your changes on Funtoo as a whole, and identify potential negative impact such as dependencies that do not resolve or build problems when your changes are active. This provides better testing that just manually using {{c|ebuild}} to test-build your changes locally. It also allows others to test your changes on ''their'' harvester systems. Be sure to leverage the #harvester Discord community to request testing of your changes as needed.
 ;Should I send a PR simultaneously for both harvester and the official release?: In general, no. If it's ready for the official release, send it there. We will periodically merge ongoing changes from the official release into harvester so they stay in-sync as possible.