View source for Funtoo:Harvester

{{Project
|summary=The harvester project is focused on providing a meta-repo for testing and active development of Funtoo Linux
|Project Category=Development
|leads=Drobbins
|members=Siris
|subpages=Branches/2022-09, Branches/2022-10, Branches/2022-11, Branches/2022-12, Branches/2023-01
|translate=no
|updates={{Project/Update
|date=2022-10-02
|status=The harvester/2022-09 branch was successfully merged into Funtoo, and the harvester/2022-10 branch is now live. Also, we now have harvester active for 1.4-release.
}}{{Project/Update
|date=2022-11-01
|status=The harvester/2022-10 branch was successfully merged into Funtoo, and the harvester/2022-11 branch is now live.
}}{{Project/Update
|date=2023-01-01
|status=The harvester/2022-12 branch was successfully merged into Funtoo, and the harvester/2023-01 branch is now live.
}}{{Project/Update
|date=2023-01-01
|status=The harvester/2022-12 branch was successfully merged into Funtoo, and the harvester/2023-01 branch is now live.
}}{{Project/Update
|date=2023-03-08
|status=The harvester/2023-01 branch was successfully merged into Funtoo, and the harvester/2023-03 branch is now live. Focus for 2023-03 will be updating Xorg.
}}{{Project/Update
|date=2023-05-30
|status=The harvester/2023-05 branch was merged into Funtoo.
}}{{Project/Update
|date=2023-07-13
|status=Harvester/2023-06, along with quite a few meson updates from geaaru/macaroniOS, have been merged into master. We will be starting our next harvester branch a bit later due to a slower-paced July. Enjoy your summer.
}}{{Project/Update
|date=2023-07-27
|status=harvester/2023-08 is available, and is primarily targeted right now towards 6.4 kernel and core toolchain updates.
}}{{Project/Update
|date=2023-10-07
|status=Harvester/2023-10 is launched. See the #gumball3000 channel on Discord for details.
}}{{Project/Update
|date=2023-11-01
|status=harvester/2023-10 is merged into Funtoo and we now have harvester/2023-11 ready for PR's!
}}{{Project/Update
|date=2024-02-01
|status=harvester/2023-11 has been merged into official Funtoo, and harvester/2024-02 has been created. Please hold off on PRs to harvester. We are going to define a more structured approach to harvester development very soon so we can move forward strategic initiatives.
}}
}}
'''To submit a PR for harvester, please send a PR via code.funtoo.org and target the {{c|harvester/2023-11}} branch of {{c|kit-fixups}}. Visit us in the #gumball3000 channel on [https://discord.gg/BNUSpUU Funtoo Discord] if you are using harvester for testing your work.'''

'''Welcome to the harvester project!'''

The harvester project is focused on providing a meta-repo that can be used for active development by Funtoo Linux contributors. This allows evaluation of experimental and potentially disruptive changes in a completely separate tree without impacting regular Funtoo Linux installs.

=== The Harvester Concept ===

Harvester exists as a branch of kit-fixups, which is listed at the top of this page.
{{c|harvester.funtoo.org}} uses it to generate a full meta-repo, containing
experimental changes. These changes are made available via git from {{c|harvester.funtoo.org}} directly, and can be consumed via {{c|ego sync}}.

=== Harvester Benefits ===

With harvester, we aim to:

;Make community testing easier: By providing a system that can be used to do full community-based integration testing without having to locally generate your own meta-repo.
;Accelerate development: By providing a 'judgement-free' space to evaluate aggressive and potentially breaking changes, and to learn how these changes impact Funtoo, for better or worse, without causing problems on live systems.
;Make things more fun: We all have limited time. Sometimes we make a 'best effort' to test our PR, but it still breaks something. This is definitely "not fun" for the contributor, for Funtoo staff, or for users. We want to avoid these situations, and avoid having to frantically rush to clean up breakages that impact users, or get upset at people for braking the tree. Harvester allows us as a community to roll out mature, tested changes to users, by testing and working out problems in harvester first. This keeps Funtoo development "fun" and a positive experience for all! We don't want the stressful way of dealing with problems to be our ''default way''. It should only be the rare exception. '''Our process should produce successful results rather than demand individual perfection.''' Harvester supports a process that keeps things "fun" by reducing the risk and thus stress related to changes to Funtoo.

=== Frequently Asked Questions ===

;What issues are you solving with harvester?: For the big picture, see [[#Harvester Benefits|Harvester Benefits]], above. On a more technical level, we need a "testing release" to test imminent changes to our releases, compatible with CI (continuous integration), to accelerate development and improve quality and test-ability of our imminent changes before they hit production systems.

;How does this relate to {{c|next}}-release in general? I thought that this was supposed to be the "experimental" branch?: {{c|next}}-release was meant to be a very up-to-date rolling release but still actually be useful for real work. So, {{c|next}}-release should not become a mechanism to integration-test potentially disruptive changes to Funtoo, nor is it a good candidate for CI to run tests. This has become more important now that {{c|next}}-release is the official recommended default install for new Funtoo Linux systems, due to it being much more current than {{c|1.4-release}}. This means that we should keep it more current, but also ''keep it functioning well'' and ''improve its reliability and usefulness''. Thus, we have a harvester {{c|next}}-release where we can test and stabilize things before they are rolled into the official {{c|next}}-release, allowing problems to be identified proactively -- before they impact production systems. This will actually '''accelerate''' development of {{c|next}}-release.

;Why not create a {{c|unstable}}-release, then? Why harvester?: Good question. Harvester is '''orthogonal''' (independent) of releases. Harvester can provide us with a CI-friendly fork of '''all''' of our releases, so it is much more flexible then a single diverging release. Creating a new {{c|unstable}}-release is not a useful option because it will diverge from our releases over time and then ultimately we will re-create the issue we are trying to solve, which is not having a CI-compatible release where we can push potentially breaking changes for manual and automated testing in a release that very closely resembles our current production releases. 

;What releases are available in harvester?: Our initial harvest had only {{c|next}} release but we are now building {{c|1.4-release}} in harvester as well. (See [[#How To Use|How To Use]], below.)

;Is there a 1.4-release harvester tree?: Yes, as of the {{c|harvester/2022-10}} branch we are supporting a harvester build of {{c|1.4-release}}.

;Why have a harvester build of 1.4-release?: The primary reason is that some of our changes can impact {{c|1.4-release}}, especially updates to ebuilds in the {{c|curated}} directory, so in order to ensure full test coverage it is prudent to add 1.4-release to harvester to determine if there is any negative impact to these staged changes.

;Is it safe to use {{c|harvester}} on production systems?: No, this is definitely not a good idea. It is purely a testing tree for Funtoo.

;Is it safe to use {{c|next}}-release on production systems?: Yes, this is fully supported and we want {{c|next}}-release to continue to improve in quality and usefulness. This is exactly why it is '''not''' a good dumping-ground for potentially breaking changes. We want to test the more aggressive updates in {{c|harvester}} first.

=== Developer FAQ ===

;Should my harvester changes go into {{c|curated}} in kit-fixups?: In general, no. We are generally testing changes to next-release in harvester, although there can be some impact to 1.4-release. In general, you should make sure that your changes only impact next-release unless you are coordinating with us on a specific issue or specifically attempting to stabilize something in 1.4-release.

;How should I coordinate with the harvester community?: Great question! You '''must''' be active in the #harvester Discord channel to talk with us there. Harvester is run by [[User:drobbins|drobbins]] (BDFL) and [[User:Adbosco|Adbosco]] (Inspector General). We each do different things. The BDFL oversees and directs the effort. The Inspector General is actively looking at the quality of various PRs and overseeing the integration testing efforts.

;<span id="Integration Testing"></span>When you say 'integration testing', what exactly do you mean?: The 'integration testing' refers to ensuring proper operation of Funtoo with your changes. We want Funtoo systems to be able to be easily updated via {{c|emerge -auDN @world}}, without problems. We want dependencies to resolve properly. We want ebuilds to build successfully. We want the updated packages to function properly. We want our official builds of our stage3's to continue to function. We want our meta-repo to regen properly, with all autogens working. All this is part of fully testing a change in Funtoo.

;What PRs should I send to harvester instead of directly to the master branch of kit-fixups?: Anything that is potentially highly disruptive and could break other packages should be sent to harvester first. This will allow your changes to become part of the harvester {{c|next}}-release (and 1.4-release, if affected) so you can see the impact of your changes on systems that are set up to use harvester (See [[#How To Use|How To Use]], below.) If your PR fixes something that is already broken (break/fix) and is very unlikely to make things worse, or adds new functionality that doesn't impact other ebuilds, it is generally still safe to send this PR directly to the official releases of Funtoo.

;I still don't understand what I should send to harvester or the official release.: OK. Then pick one, giving preference to harvester. If you send your PR to the official release and we determine that it is potentially disruptive or could break other ebuilds (or if it is conspicuously missing any documentation of testing performed, or we find the testing perfomed to be inadequate,) we will identify this, decline the PR, and ask you to re-send it to harvester. If you send it to harvester, we will likely approve it so you can see the results of your changes when harvester is regenned. You can then continue integration-testing your changes in harvester until they are ready to be released. At this point, please send a PR for the official release and document the testing you performed in the PR description.

;How will harvester be kept in-sync with the official releases?: During development, they will temporarily diverge, although we will frequently (several times per week) merge in official updates to the harvester branch to keep harvester in-sync as possible with the official release. But to prevent harvester continually getting more and more out-of-sync with the official release, we will periodically create a kit-fixups branch like {{c|harvester/2022-10}} that is always forked from the official upstream releases of Funtoo. So periodically, we re-sync with the official release by creating a new timestamped harvester branch, before adding experimental changes. The frequency is something we can control and we will determine a cadence that works for harvester users from practical experience.

;How will changes in harvester be added to official releases?: Again, we have options here. If we have a bunch of experimental changes in harvester, and they all get worked on and become quite reliable, we can directly merge the harvester branch into master. Or, if only some changes become stable, the ones that are stable can be sent as PR's into master (the official releases.) Or, if we try a bunch of things that are a total disaster, we can abandon the timestamped harvester branch and create a new one. So we have the ability to adapt as needed and don't need to have a 'set in stone' git flow.

;What if there are still in-progress changes I'm working on in harvester when you fork a new timestamped release of harvester from the official release?: Ah, the importance of coordination. Be available in the #harvester channel on Discord so you can be aware of our cadence. We will either coordinate with you there or you can submit a PR with your work so far for the new harvester, if your testing requires more than a single "cycle" of harvester to be completed.

;How does this work with {{c|gnome-kit}} and other ''sourced'' kits which have their own git repositories, such as {{c|gnome-kit-sources}}?: We are working on a script that will create harvester branches for all sourced kits, so very soon you will have a {{c|harvester/2022-10/3.36-prime}} branch in {{c|gnome-kit-sources}} which can receive experimental GNOME changes. For now, we can create these branches manually as needed.

;How do I "integration-test" my harvester changes?: See [[#How To Use|How To Use]], below. You will set up a Funtoo system (or container, using [[LXD]]) and set it up to use harvester at its meta-repo, so that {{c|ego sync}} will grab harvester rather than our official releases. Then you can see how your system behaves with your changes (see [[#Integration Testing|Integration Testing]]). By doing this, you can see the impact of your changes when integrated fully into Funtoo, and be able to identify potential negative impact such as dependencies that do not resolve or build problems when your changes are active and a world update is performed. This provides better testing that just manually using {{c|ebuild}} to test-build your changes locally. It also allows others to test your changes on ''their'' harvester systems. CI is coming soon. Be sure to leverage the #harvester Discord community to request testing of your changes as needed.

;How do I configure a FFS stage1 to use Harvester for as its ego sync url?: If you are building Funtoo From Scratch stage1s and want to use the latest Harvester branch there is an experimental patch for the {{c|profiles/gnu/steps/fchroot.yaml}} config file. See the [https://code.funtoo.org/bitbucket/projects/CORE/repos/ffs/browse/README.rst?at=refs%2Fheads%2Fharvester%2F2022-11 README.rst's Harvester Testing section] for the full documentation.

;How do I configure Metro to use Harvester for its portage snapshot during stage2 and stage3 builds?: Before running any metro {{c|ezbuild.sh}} or automated metro commands to trigger stage2/3 builds, simple export this shell environment variable to ensure the ego sync generates a Harvester portage snapshot from the latest Harvester branch:

{{file|name=EGO_SYNC_BASE_URL|lang=bash|desc=ego sync bash url env var|body=
export EGO_SYNC_BASE_URL="git://harvester.funtoo.org/funtoo/tree/{repo}"
}}

You can also add this environmental shell variable export to root user's {{c|.bashrc}} , which will ensure it gets sourced in {{c|.bash_login}} and is always defined for the root user.

This is actually a feature of ego -- this variable is detected and utilized by ego sync when set. The {{c|{repo}}} is a literal string and should appear as-is.  Same format as ego.conf's sync_base_url setting.

;Should I send a PR simultaneously for both harvester and the official release?: In general, no. If it's ready for the official release, send it there. We will frequently merge ongoing changes from the official release into harvester so they stay in-sync as much as possible.

;Do I need to worry about "blowing up" harvester with a really bad PR?: Most of the time, no, you don't need to be extremely concerned about this, although we ask that your PRs have undergone a reasonable level of local testing and you don't send any "crap shot" / "I'm feeling lucky" / "this should work" PR's to harvester as it can impede the experience for other harvester users. If your PR contains a broken autogen which prevents the tree from regenning, it may be reverted and we will reach out to you in the #gumball3000 Discord channel to let you know.

;Because I am using harvester, does this mean I don't need to worry about my PR's breaking the official release?: That would be unwise. We '''always''' recommend you are available to identify any missed issues in your changes after they go live in the official tree. Harvester will greatly reduce -- but not eliminate -- the potential of breakage in our live tree. Harvester helps us to be proactive about finding problems. Ultimately, it rests on the contributor -- with Funtoo staff as an emergency fallback -- to identify and fix any breakage on user systems caused by our changes.

=== How To Use ===

Currently, harvester provides an experimental next-release tree and 1.4-release tree.

To use harvester on an existing next-release system, add the following lines to your
{{f|/etc/ego.conf}}:

{{file|name=/etc/ego.conf|body=
[global]
sync_base_url = git://harvester.funtoo.org/funtoo/tree/{repo}
}}

or:

{{console|body=
# ##i##echo 'sync_base_url = git://harvester.funtoo.org/funtoo/tree/{repo}' >> /etc/ego.conf
}}

Then, back up your existing meta-repo, and re-sync the new harvester meta-repo:

{{console|body=
# ##i##cd /var/git
# ##i##mv meta-repo meta-repo.bak
# ##i##ego sync
}}

Your system will now have the harvester next-release meta-repo available to it,
so you can test the latest experimental changes in harvester via {{c|emerge}}.

=== How To Submit ===

As with all PR's, it is required that you open an issue on {{c|bugs.funtoo.org}}. Once
this is done, and the issue has been triaged so that is no longer in the "Intake" state,
it is possible to start progress on the issue and submit a PR referencing the issue, 
targeting the kit-fixups branch listed at the top of the page. The PR will be on an
accelerated track to be merged into harvester. When the harvester tree is next regenerated, the change will appear in the tree and {{c|ego sync}} will make it available on any Funtoo Linux systems using harvester.

Once your changes have been sufficiently tested, it is then possible to submit a vetted
PR to the production next-release tree, by submitting a PR to the kit-fixups master
branch.

=== Using Sourced Kits ===

Most kits can receive new ebuilds and autogens via kit-fixups, and thus it is easy to add experimental ebuilds and autogens
to most kits, but there are exceptions.

Any "sourced" kits are kits that are fully self-contained in their own repos, such as
gnome-kit-sources. You can find a full list of sourced kits by looking at [https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse/releases/next/repositories.yaml#147 kit-fixups/browse/releases/next/repositories.yaml] -- any kit with {{c|kind: sourced}}
is a sourced kit. This means that kit-fixups isn't used to "tweak" the contents of the kit -- instead, the ebuilds and autogens
come from the referenced branch as-is, and all updates appear in this separate "sources" repo and branch.

This creates a problem for harvester -- how do you inject experimental changes into sourced kits? The answer is to coordinate with
our master of the harvest so that we can create a special harvester branch on the sourced kit for you, and update our YAML so that
this harvester branch is used for our harvester tree. Basically, we will create a {{c|harvester/YYYY-MM}} branch in the sourced kit,
and then you submit the PR against that. We don't do this automatically since many kits are not sourced and the creation of these
branches is not yet automated.

{{Note|We will probably automate the creation of harvester branches for sourced kits in the near future, and add details here about
how to submit to harvester for sourced kits.}}

=== How Master Changes are Merged into Harvester ===
Ensure the following setting exists in the local kit-fixups git configuration
{{f|.git/config}}:

{{file|name=.git/config|body=
[remote "upstream"]
	url = ssh://git@code.funtoo.org:7999/core/kit-fixups.git
	fetch = +refs/heads/*:refs/remotes/upstream/*
}}

Frequently (several times a week), I will merge recent changes in the master (production) kit-fixups branch into harvester,
so that harvester stays in sync with git master. This process uses the following commands:

{{console|body=
$ ##i##cd kit-fixups
$ ##i##git checkout master
$ ##i##git pull
$ ##i##git checkout harvester/2022-10
$ ##i##git pull
$ ##i##git merge master
\# Push to your fork and to the production tree:
$ ##i##git push
$ ##i##git push upstream
}}

=== How Harvester is Merged into Master ===

Ensure the following setting exists in the local kit-fixups git configuration
{{f|.git/config}}:

{{file|name=.git/config|body=
[remote "upstream"]
	url = ssh://git@code.funtoo.org:7999/core/kit-fixups.git
	fetch = +refs/heads/*:refs/remotes/upstream/*
}}

With {{c|kit-fixups}}, the following approach should be used to merge the harvester branch into master. We originally used "squashed" commits --
however, this deletes the proper attribution of the original changes and mis-attributes them to the person doing the squash. Therefore, this
updated method should be used:

{{console|body=
$ ##i##cd kit-fixups
$ ##i##git checkout master
$ ##i##git clean -fd
$ ##i##git reset
$ ##i##git pull
$ ##i##git checkout -b merge/harvester/2024-02
Switched to a new branch 'merge/harvester/2024-02'
$ ##i##git merge harvester/2024-02
$ ##i##cd releases
$ ##i##git log --stat .
(Take note of the SHA1 of the commit that updates the release.yaml to reference harvester)
$ ##i##git revert <sha1>
$ ##i##git checkout master
$ ##i##git merge merge/harvester/2024-02
(Push to your fork and, optionally, to the production tree:)
$ ##i##git push
$ ##i##git push upstream
}}

=== Harvester Tree Regeneration ===

In order to get kit-fixups ready to generate harvester, a small change to the release.yaml is needed, which is done in the harvester branch of kit-fixups that you are using:

{{file|name=releases/next/repositories.yaml|lang=yaml|body=
  # Have all of our remote configurations defined in one place for ease of maintenance
  remotes:
    dev:
      url: https://code.funtoo.org/bitbucket/scm/auto/{repo}
    prod:
      url: ssh://repos@harvester.funtoo.org/funtoo/tree/{repo}
}}

This change above, when committed, allows {{c|merge-kits}} to properly identify the upstream git URL to use for kits. Typically, the {{c|--prod}} option is used for tree regen, so in this case, adjust the {{c|prod/url}} to reflect the location of your harvester upstream URL where the kits will live. Without {{c|--prod}}, the trees can be stored locally and not pushed up to a remote server.

You now have to tell harvester to use this new branch for tree regeneration. You can do it in the   
file {{c|/home/tree/harvester.sh}} inside of the lxd container {{c|harvester}} on {{c|lxd-infra-1.funtoo}} host. Set the {{c|HARVEST}} to the wanted value.

{{file|name=/home/tree/harvester.sh|lang=bash|desc=set the HARVEST value|body=
#!/bin/sh
#
#set -x
RELEASE=$1
HARVEST="2023-01"
if [ -z "$RELEASE" ]; then
        echo "Please specify release."
        exit 1
fi
export PATH=~/development/funtoo-metatools/bin:$PATH
export PYTHONPATH=~/development/funtoo-metatools
hash -r
set -e
set -o pipefail
LOGPATH=$HOME/log/
install -d $LOGPATH
( cd $HOME/development/funtoo-metatools; git pull )
time ( merge-kits ${RELEASE} --prod --fixups_branch harvester/${HARVEST} --fixups_url=ssh://git@code.funtoo.org:7999/core/kit-fixups.git --create_branches 2>&1 | tee "$LOGPATH/${LOGFILE:-output-$(date +%F_%T)_log.txt}" )
}}

=== Harvester Troubleshooting ===

Harvester application is running in the lxd container {{c|harvester}} on {{c|lxd-infra-1.funtoo}} host. To connect to the harvester container you should have a running [[Funtoo:User Services/VPN Setup|WireGuard VPN Setup]].
{{ProjectFooter}}