注意：

The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.

Difference between revisions of "Funtoo:Metatools/Advanced Usage/Gitolite Setup"

From Funtoo
Jump to navigation Jump to search
 
(20 intermediate revisions by 2 users not shown)
Line 2: Line 2:
== Overview ==
== Overview ==


For our local development setup, we will be using [http://gitolite.com gitolite]. Gitolite will make things quite a bit easier by managing git repositories for us. ''Think of gitolite as your own private GitHub that has no Web user interface'' (we modify its settings by pushing to its special {{c|gitolite-admin}} repo) and you'll have a pretty good idea of what gitolite does. We will be using the following systems in these examples:
For our local development setup, we will be using [http://gitolite.com gitolite]. Gitolite will make things quite a bit easier by managing git repositories for us. ''Think of gitolite as your own private GitHub that has no Web user interface'' (we modify its settings by pushing to its special {{c|gitolite-admin}} repo) and you'll have a pretty good idea of what gitolite does. To keep things simple, we will be using a single development system in our setup example below. However, because gitolite can be accessed over the network, you can easily set up gitolite on another system on your LAN or a remote system on the Internet and securely access it simply by modifying the git URL.
 
* {{c|repohost}} - this system will be running gitolite under the {{c|repos}} user account and will house git repositories for meta-repo and kits so that they are stored at a handy central location. If you are going to be using your dev workstation as your "repohost", you can simply replace all references to {{c|repohost}} with {{c|localhost}} below :)
* {{c|ryzen}} - in these examples, this will be the primary development workstation, which will be used for editing cloned git code as well as generating custom kits. Once generated, the custom meta-repo and kits are pushed up to {{c|repohost}}.
 
{{Note|When you follow this guide, it is certainly possible to have {{c|repohost}} and {{c|ryzen}} be the same computer. If you set things up this way, simply use {{c|localhost}} instead of {{c|repohost}} in all command-line steps and configuration files.}}


{{Important|This document assumes you have basic knowledge of {{c|ssh-keygen}} and how to generate public/private SSH key pairs. If you don't know how to to this, see [[Funtoo Containers#Generating SSH Keys|Generating SSH Keys]] for quick steps or [[OpenSSH Key Management, Part 1]] for a more detailed introduction. For this article, you'll probably want to generate a private keys without a passphrase, which is more convenient but a much greater security risk if the private key gets compromised, or one with a passphrase but using [[keychain]] to manage ssh-agent for you.}}
{{Important|This document assumes you have basic knowledge of {{c|ssh-keygen}} and how to generate public/private SSH key pairs. If you don't know how to to this, see [[Funtoo Containers#Generating SSH Keys|Generating SSH Keys]] for quick steps or [[OpenSSH Key Management, Part 1]] for a more detailed introduction. For this article, you'll probably want to generate a private keys without a passphrase, which is more convenient but a much greater security risk if the private key gets compromised, or one with a passphrase but using [[keychain]] to manage ssh-agent for you.}}
Line 15: Line 10:
=== Installation ===
=== Installation ===


To set up gitolite on your LAN, first choose a system that will be used to house your meta-repo and kits git repositories. You can do this on the same system you will be using for testing (and even development), or you can set it up on a dedicated system. It's actually fine to set this up anywhere on the Internet, as git will use ssh to access this repository, but for the purposes of this article, we're assuming you're setting it up somewhere on your LAN.  We will refer to this system as '''repohost'''.  
To set up gitolite, we will use a {{c|repos}} user on your local development system. On this system, perform the following steps as root:
 
On this system, perform the following steps as root:


{{console|body=
{{console|body=
# ##i##useradd -m repos
%repohost% ##i##useradd -m repos
}}
}}


The {{c|repos}} user will be a dedicated user account on the system that will have gitolite enabled and will house our git repositories. Now, we are going to {{c|su}} to this new user on '''repohost''' and perform gitolite configuration:
The {{c|repos}} user will be a dedicated user account on the system that will have gitolite enabled and will house our git repositories. Now, we are going to {{c|su}} to this new user and perform gitolite configuration:


{{console|body=
{{console|body=
# ##i##su --login repos
%repohost% ##i##su --login repos
$ ##i##git clone https://github.com/sitaramc/gitolite
$ ##i##git clone https://github.com/sitaramc/gitolite
$ ##i##install -d ~/bin
$ ##i##install -d ~/bin
}}
}}


Now, as the {{c|repos}} user, add the following to the end of your {{c|~/.bashrc}} file:
Now, as the {{c|repos}} user, add the following within your {{c|~/.bashrc}} file:


{{file|name=~/.bashrc|body=
{{file|name=~/.bashrc|body=
Line 44: Line 37:
}}
}}


Now, your {{c|repos}} account is almost ready to be used for hosting repositories. The way gitolite works is that it is going to basically take over ssh access to the account, so that when you connect via ssh with git, it will perform its own authentication. For this to work, you will need to enable your own "master key" to access gitolite. To do this, you'll want to decide from which account you'll want to administer gitolite itself. I prefer to use my "drobbins" account on my development workstation '''ryzen''', so I will copy my ssh public key from {{c|~/.ssh/id_rsa.pub}} to {{c|/var/tmp/ryzen-drobbins.pub}} on the gitolite system, and then perform the following steps to "prime" gitolite with this admin public key:
Now, your {{c|repos}} account is almost ready to be used for hosting repositories. The way gitolite works is that it is going to basically take over ssh access to the account, so that when you connect via ssh with git, it will perform its own authentication. For this to work, you will need to enable your own "master key" to access gitolite.  
 
To do this, you'll want to decide from which account you'll want to administer gitolite itself. I prefer to use my "drobbins" account on local system, so I will copy my ssh public key from {{c|~/.ssh/id_ed25519.pub}} to {{c|/var/tmp/drobbins.pub}}, and then perform the following steps to "prime" gitolite with this admin public key -- do this as the {{c|repos}} user:


{{console|body=
{{console|body=
$ ##i##gitolite setup -pk /var/tmp/ryzen-drobbins.pub
$ ##i##gitolite setup -pk /var/tmp/drobbins.pub
}}
}}


Gitolite will now be initialized to recognize the {{c|drobbins}} remote account as an administrator, which will allow this remote account to clone from {{c|repos@repohost:gitolite-admin}} and push any changes to this special git repository which contains the master configuration for gitolite. This is important because we will be performing the rest of gitolite setup over ssh, using this account.
Gitolite will now be initialized to recognize the {{c|drobbins}} account as an administrator, which will allow this account to clone from {{c|repos@localhost:gitolite-admin}} and push any changes to this special git repository which contains the master configuration for gitolite. This is important because we will be performing the rest of gitolite setup over ssh, using this account.


{{Note|OK, gitolite is installed on '''repohost'''! From this point forward, we will be using the {{c|drobbins}} (or equivalent) account on your development workstation to configure gitolite remotely.}}
{{Note|OK, gitolite is installed! From this point forward, we will be using the {{c|drobbins}} (or equivalent) account on your development workstation to configure gitolite.}}


{{Note|If you are setting up gitolite on a separate server, and assuming you have ssh properly configured, it would be prudent at this point to test the connection to the remote server from your local workstation . Perform {{c|ssh -T}}, or ssh test command appropriate to your setup, as shown below to verify you can connect:
{{Note|If you are setting up gitolite on a separate server, and assuming you have ssh properly configured, it would be prudent at this point to test the connection to the remote server from your local workstation . Perform {{c|ssh -T}}, or ssh test command appropriate to your setup, as shown below to verify you can connect:
Line 80: Line 75:
=== gitolite-admin Clone ===
=== gitolite-admin Clone ===


Now that gitolite is ready on '''repohost''', we can do everything else remotely. I am going to use the {{c|drobbins}} account on my development workstation '''ryzen''', and you will use whatever account is associated with the public key you loaded into gitolite. I like storing my development repos in {{c|/var/src}} on '''ryzen''', so I'll go ahead and clone the gitolite-admin repo to that location so it can live along all my other git repos. Feel free to put this git repo wherever you like to store git repos that you develop on:
Now that gitolite is ready under the {{c|repos}} user on your local system, we can configure the rest by cloning the {{c|gitolite-admin}} repo and commiting configuration changes to it. I am going to use the {{c|drobbins}} account on the same system to do this, and you will use whatever account is associated with the public key you loaded into gitolite. I like storing my development repos in {{c|~/development}} , so I'll go ahead and clone the gitolite-admin repo to that location so it can live along all my other git repos. Feel free to put this git repo wherever you like to store git repos that you develop on:


{{console|body=
{{console|body=
$ ##i##cd /var/src
$ ##i##cd ~/development
$ ##i##git clone repos@repohost:gitolite-admin
$ ##i##git clone repos@localhost:gitolite-admin
$ ##i##cd gitolite-admin
$ ##i##cd gitolite-admin
}}
}}
Line 92: Line 87:
=== gitolite Configuration ===
=== gitolite Configuration ===


Since I will be generating meta-repo and kits on '''ryzen''', this system will need to have permissions to create repositories in gitolite. Although I do my regular development as the drobbins user, I will be running the meta-repo generating script on '''ryzen''' as root, so the root user on ryzen needs to be granted access to gitolite. I would typically do this on '''ryzen''' as follows. First, as root, grab root's public ssh key:
Since I will be generating meta-repo and kits using my {{c|drobbins}} user, this account will need to have permissions to create repositories in gitolite.  
 
We're ready to edit {{c|conf/gitolite.conf}} so that it looks like this:
{{console|body=
# ##i##cp /root/.ssh/id_rsa.pub /var/tmp
}}
 
{{Note|If {{f|/root/.ssh/id_rsa.pub}} doesn't exist, you can create it by typing {{c|ssh-keygen -t rsa}} as root and specifying no passphrase for encryption.}}
 
Then, as my regular {{c|drobbins}} user that I typically use to perform development, I will install root's public key in my cloned gitolite repository. This is the first step to grant
the ryzen root user direct access to gitolite:
 
{{console|body=
$ ##i##cd /var/src/gitolite-admin
$ ##i##cp /var/tmp/id_rsa.pub keydir/ryzen-root.pub
$ ##i##git add keydir/*
}}
 
{{Note|It's important to change the filename of the public keys you are adding to the {{c|gitolite-admin}} repository. I typically use the format {{c|host-user.pub}}.}}
 
Now, we're ready to edit {{c|conf/gitolite.conf}} so that it looks like this:


{{file|name=gitolite.conf|body=
{{file|name=gitolite.conf|body=
# Group definitions below, starting with @. This makes it easy to associate multiple ssh keys with a particular person.
# Group definitions below, starting with @. This makes it easy to associate multiple ssh keys with a particular person.


@drobbins = ryzen-drobbins
# Admins will have full control over gitolite.
@repomgr = ryzen-root
@admins = drobbins


# To enable read-only access to your meta-repo and kits, use this along with
# Developers will be able to create, read and write git repositories.
# commented-out line under wildrepo. You will need to add box1-root.pub and
@developers = drobbins
# box2-root.pub to keydir/ as well. This is good for boxes that will be testing
# your meta-repo and kits only but should not be able to modify them.


#@reporead = box1-root box2-root
# Users will have read-only access.
@users = drobbins


# repositories:
# repositories:


# SPECIAL ADMIN REPO BELOW -- modify with care! I've switched over to using the @drobbins group instead of
# SPECIAL ADMIN REPO BELOW -- modify with care!  
# referencing the individual ryzen-drobbins key directly.


repo gitolite-admin
repo gitolite-admin
     RW+    =  @drobbins
     RW+    =  @admins


# AUTO-CREATED (wild) REPOS: gitolite will auto-create repos under wildrepo/ for us
# AUTO-CREATED (wild) REPOS: gitolite will auto-create repos under wildrepo/ for us
Line 138: Line 113:


repo wildrepo/..*
repo wildrepo/..*
     C      =  @repomgr
     C      =  @developers
     RW+    =  @repomgr @drobbins
     RW+    =  @developers
# NOTE: to enable read-only access for certain boxes, uncomment this line:
    R      =  @developers @users
#  R      =  @reporead
}}
}}


Line 160: Line 134:
$
$
}}
}}


== Configuration ==
== Configuration ==


{{Important|This section is somewhat out-of-date and is in the process of being updated. The current tool used for generating meta-repo is metatools, setup of which is [https://projects.funtoo.org/metatools/docs/install.html documented here]. Because metatools was based on merge-scripts, the configuration is nearly identical and besides [https://projects.funtoo.org/metatools/docs/install.html initial setup of metatools], the following instructions should generally apply.}}
Now, as your own user account, on your development system (which could be the same Funtoo system), you will want to {{c|emerge metatools}} or use git to clone the master branch of metatools. Here's how we would use the master branch:


Now that merge-scripts is cloned, we will need to create a {{c|~/.merge}} configuration file. Here is an example file that can serve as a starting point. In this example, developer {{c|bcowan}} has forked the kit-fixups repository on code.funtoo.org, and wants to test his personal changes by generating a complete meta-repo of his own:
{{console|body=
$ ##i##emerge --onlydeps metatools
$ ##i##mkdir development
$ ##i##cd development
$ ##i##git clone https://code.funtoo.org/bitbucket/scm/~drobbins/funtoo-metatools.git
}}


{{file|name=/root/.merge|body=
If using the git master version, you will need to add the following to your {{c|~/.bashrc}} :
[sources]


flora = https://code.funtoo.org/bitbucket/scm/co/flora.git
{{file|body=
kit-fixups = ssh://git@code.funtoo.org:7999/~bcowan/kit-fixups.git
export PYTHONPATH=~/development/funtoo-metatools
gentoo-staging = https://code.funtoo.org/bitbucket/scm/auto/gentoo-staging.git
export PATH=~/development/funtoo-metatools:$PATH
}}


[destinations]
== Create Your Own Kit-Fixups ==


base_url = repos@repohost:wildrepo/staging
You will now want to clone kit-fixups from code.funtoo.org, and create a commit that updates {{f|~/kit-fixups/releases/next/repositories.yaml}} to adjust the remotes as follows:


[branches]
{{file|name=releases/next/repositories.yaml|lang=yaml|body=
 
  # Have all of our remote configurations defined in one place for ease of maintenance
flora = master
  remotes:
kit-fixups = master
    dev:
meta-repo = master
      url: repos@gitolitehost:wildrepo/staging/{repo}
    prod:
      url: repos@gitolitehost:wildrepo/staging/{repo}
}}
}}


==== Sources Section ====
Push this commit up to your personal kit-fixups.
 
Let's walk through this configuration file. The {{c|[sources]}} section defines locations of repositories that the merge scripts will use as sources for creating kits and meta-repo. In the above sample config, we are using the official [[Flora]] repository from Funtoo, and the official gentoo-staging repository used by Funtoo, but we are using our own fork of [https://code.funtoo.org/bitbucket/projects/CORE/repos/kit-fixups/browse kit-fixups], which will allow us to add new ebuilds that will appear in kits, such as bug fixes to existing ebuilds in kits, as well as security fixes. For a core Funtoo Linux developer, this is a good way to start. If you are more interested in contributing third-party ebuilds, then you may instead choose to create your own fork of [https://code.funtoo.org/bitbucket/projects/CO/repos/flora/browse flora], and use our standard kit-fixups repository. Or, you could choose to create forks of both. The recommended best practice is to use our upstream repos when possible, and fork only those repos you want to customize. This way, you'll ensure that you have the most up-to-date versions of ebuilds in those unforked repos.
 
==== Branches Section ====
 
The {{c|[branches]}} section is used to define the default branches that are used by the merge-scripts. In general, sticking with {{c|master}} is fine, but if you need the flexibility, you can point the merge scripts to a particular feature branch to use instead, for example.
 
==== Work Section ====
 
The {{c|[work]}} section is used to define paths where the merge-scripts will do their work. The {{c|source}} and {{c|destination}} settings above are good defaults, and define where the merge-scripts will clone source repositories and destination (written to) repositories, such as kits and meta-repo. These are kept in two separate hierarchies so they don't get mixed up.


== Generating New Kits ==
== Generating New Kits ==


With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be stored on '''repohost'''. Here are the steps you'd perform:
With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be pushed up to '''repos@localhost/staging/repo_name'''. The '''next''' is the release of Funtoo.


{{console|body=
{{console|body=
 
$ ##i##merge-kits next --prod --fixups_url=ssh://git@code.funtoo.org:7999/myusername/kit-fixups.git --create_branches
# ##i##cd /root/merge-scripts
# ##i##bin/merge-all-kits 1.3-release
}}
}}


Before starting the script for the first time, you should emerge jinja and configure your git user.name and user.email variables.


{{console|body=
{{c|merge-kits}} will proceed to create new kits and meta-repo, and will push them up to '''repos@localhost:wildrepo/staging/meta-repo''', '''repos@localhost:wildrepo/staging/core-kit''', etc. This process can take quite a while but has been optimized to run quickly on multi-core systems and use caching extensively so successive runs will complete quickly.
# ##i##emerge -u jinja
# ##i##git config --global user.email "you@example.com"
# ##i##git config --global user.name "Your Name"
}}
 
{{c|merge-all-kits}} will proceed to create new kits and meta-repo, and will push them up to repos@repohost:wildrepo/staging/meta-repo, repos@repohost:wildrepo/staging/core-kit, etc. This process can take quite a while but has been optimized to run quickly on multi-core systems.


== Using New Kits ==
== Using New Kits ==
Line 225: Line 186:
[global]
[global]


sync_user = root
sync_user = drobbins
sync_base_url = repos@repohost:wildrepo/staging/{repo}
sync_base_url = repos@gitolitehost:wildrepo/staging/{repo}


# Yes, you are supposed to have a literal "{repo}", above. Ego recognizes this special pattern.
# Yes, you are supposed to have a literal "{repo}", above. Ego recognizes this special pattern.
Line 233: Line 194:
}}
}}


You will want to make sure that whatever system is connecting to '''repohost''' is permitted by gitolite, and has its {{c|/root/.ssh/id_rsa.pub file}} stored and committed to the gitolite-admin repository's {{c|keydir/}} and is referenced in {{c|conf/gitolite.conf}}. Otherwise, gitolite will not allow this system to connect. Of course, if you are configuring this on the system that's running {{c|merge-all-kits.py}}, it already has RW permissions to these repositories. Otherwise, you will want to make sure that gitolite has the system's keys as part of its {{c|@reporead}} group.
Since we are syncing as a regular user, we will need to make sure that our user account is in the {{c|portage}} group and that /var/git is writable by, in this case, {{c|drobbins}}:


{{Note|We change the {{c|sync_user}} to {{c|root}} to force {{c|ego sync}} to use {{c|/root/.ssh/id_rsa}} for authentication. By default, {{c|ego sync}} will attempt to use the {{c|portage}} user which does not have a private key installed, and thus will not be able to authenticate with gitolite. Rather than mess with the {{c|portage}} user and give it a proper home directory and ssh key pair, it's easier just to not drop perms to {{c|portage}} in the first place.}}
{{console|body=
# ##i##chmod g+rwx /var/git
}}


Now, let's move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run {{c|ego sync}}:
Let's also move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run {{c|ego sync}}:


{{console|body=
{{console|body=
# ##i##cd /var/git
# ##i##cd /var/git
# ##i##mv meta-repo meta-repo.official
# ##i##mv meta-repo meta-repo.official
# ##i##ego sync
# ##i##exit
Syncing xorg-kit branch 1.20-release
$ ##i##ego sync
Cloning into '/var/git/meta-repo/kits/xorg-kit'...
Running as regular user.
Initialized empty Git repository in /home/repos/repositories/wildrepo/staging/xorg-kit.git/
Syncing meta-repo
fatal: Remote branch 1.20-release not found in upstream origin
Cloning into '/var/git/meta-repo'...
##r##ERROR: Could not clone kit 'xorg-kit' into '/var/git/meta-repo/kits/xorg-kit'.
X11 forwarding request failed on channel 0
fatal: the remote end hung up unexpectedly
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 13 (delta 1), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (13/13), done.
Resolving deltas: 100% (1/1), done.
Syncing browser-kit branch 1.4-release
}}
}}


Ego will now sync your custom repository, ''but it will fail.''. This is OK -- it is failing because gitolite does not have local copies of independently-maintained (ie. not auto-generated) kits. To fix this, you can either create mirrors of the independent kits on your gitolite, or alternatively perform this simple work-around. Go to https://code.funtoo.org/bitbucket/projects/INDY and for each kit listed there, clone to {{f|/var/git/meta-repo/kits}} as root, as follows:
If you type {{c|emerge -auDN @world}}, ego will now be using your custom kits, rather than the official Funtoo ones. This means that you can perform a variety of things you couldn't before. You can now add your own custom ebuilds to your fork of {{c|kit-fixups}}, and {{c|merge-kits}} will automatically incorporate these changes into your own custom kits. This will allow you to locally test any changes before submitting them as pull requests to Funtoo. You will also be able to maintain your own meta-repo and kits with your own local modifications, and have your systems use these meta-repo/kits instead of the official Funtoo ones.


{{console|body=
== Using New Kits with Metro ==
# ##i##cd /var/git/meta-repo/kits
 
# ##i## git clone https://code.funtoo.org/bitbucket/scm/indy/gnome-kit.git
Rather than using new kits on a local Funtoo Linux system, you may want to use the kits to perform a metro build. This is desirable when you may want to use metro to perform [[Wikipedia:Continuous integration|continuous integration]] by attempting to perform a full stage1, 2 and 3 build, which is what the steps in this example accomplish, or simply test some hypothetical changes to the tree
# ##i## git clone https://code.funtoo.org/bitbucket/scm/indy/python-kit.git
before creating a PR. Let's look at how to do this.
... etc ...
}}


Once this is complete, try an {{c|ego sync}} again, and with local copies of the independent kits it should all work:
First, since metro runs as root, you will need to ensure that the {{c|root}} user on your local system has read access to gitolite. Perform the following steps:


{{console|body=
{{console|body=
# ##i##ego sync
# ##i##cp /root/.ssh/id_ed25519.pub /var/tmp/root.pub
...
# ##i##exit
##g##Sync successful and kits in alignment! :)
$ ##i##cd ~/development/gitolite-admin/keydir
Updating /etc/portage/repos.conf...
$ ##i##cp /var/tmp/root.pub .
Updating profiles at /etc/portage/make.profile/parent...
$ ##i##git add *
}}
}}


If you type {{c|emerge -auDN @world}}, ego will now be using your custom kits, rather than the official Funtoo ones. This means that you can perform a variety of things you couldn't before. You can now add your own custom ebuilds to your fork of {{c|kit-fixups}}, and {{c|merge-all-kits.py}} will automatically incorporate these changes into your own custom kits. This will allow you to locally test any changes before submitting them as pull requests to Funtoo. You will also be able to maintain your own meta-repo and kits with your own local modifications, and have your systems use these meta-repo/kits instead of the official Funtoo ones.
Now, give the {{c|root.pub}} key the ability to have read-only access to the repository (showing only the changes required):
 
== Using New Kits with Metro ==
 
Rather than using new kits on a local Funtoo Linux system, you may want to use the kits to perform a metro build. This is desirable when you may want to use metro to perform [[Wikipedia:Continuous integration|continuous integration]] by attempting to perform a full stage1, 2 and 3 build, which is what the steps in this example accomplish. To do this, first set up metro on your system:


{{console|body=
{{file|name=~/development/gitolite-admin/conf/gitolite.conf|body=
# ##i##cd /root
@users = drobbins root
# ##i##git clone https://code.funtoo.org/bitbucket/scm/core/metro.git
# ##i##cd metro
# ##i##scripts/autosetup
}}
Now, we will set the {{c|ego}}, administration tool of Funtoo/Linux. The way it is used with metro is independent from {{c|app-admin/ego}} installed on your box. Setup is easy as follows:
{{console|body=
# ##i##cd /root
# ##i##git clone https://code.funtoo.org/bitbucket/scm/core/ego.git
}}
}}
This way you will have {{c|/root/ego}} directory with {{c|ego}} binary that is then used by metro.


After following the interactive prompts, you'll then want to perform the following steps to install an ego.conf for use by metro:
Now, commit changes:


{{console|body=
{{console|body=
# ##i##install -d /etc/metro/chroot/etc
$ ##i##git add .
# ##i##vim /etc/metro/chroot/etc/ego.conf
$ ##i##git commit -a -m "gitolite config updates"
}}
}}


To ensure that ego uses your custom local meta-repo, you'll want to ensure that the following lines are in your /etc/metro/chroot/etc/ego.conf:
Now, the root user, which is used by Metro to perform the {{c|ego sync}}, should be able to connect to the repository.


{{file|body=
Next, we will want to instruct Metro to grab the custom meta-repo instead of the official one. To do this, first set up [[Metro]] on your system.
[global]
Then, prior to running metro, set the {{c|EGO_SYNC_BASE_URL}} in your environment as follows:
 
sync_user = root
sync_base_url = repos@repohost:wildrepo/staging/{repo}
}}
 
Without these ego options, metro will use the official meta-repo instead of your local custom meta-repo. Of course, be sure to specify any custom kit branches to use as well.
 
Once your ego.conf is installed, you can perform a metro build with something similar to the following commands:


{{console|body=
{{console|body=
# ##i##cd /root/metro
# ##i##export EGO_SYNC_BASE_URL="repos@localhost:wildrepo/staging/{repo}"
# ##i##scripts/ezbuild.sh funtoo-current x86-64bit intel64-haswell full
}}
 
For a full continuous integration loop, which includes regeneration of new meta-repo contents, the following script can be used:
 
{{file|lang=bash|body=
#!/bin/bash
/root/merge-scripts/bin/merge-all-kits push
rm /home/mirror/funtoo/funtoo-current/snapshots/portage-*
/root/metro/scripts/ezbuild.sh funtoo-current x86-64bit intel64-haswell full
}}
}}


The extra "rm" command is included to clean out any previous daily snapshots, to force metro to regenerate snapshots. Otherwise, any existing daily portage snapshots will be used and new ones will not be generated.
Metro should grab these settings and automatically adjust the {{c|ego.conf}} it uses based on these settings, to grab your meta-repo instead of the official one. You should now be able to perform
 
local or CI testing of your changes. It's always a good idea to put this setting in {{c|~/.bashrc}} of a CI user account so that it is always set correctly.
{{Important|If you are "switching back and forth" between different versions of kits, it may be necessary to wipe metro's binary package cache to avoid build failures caused by emerge trying to install binary packages built for a different environment. A symptom of a package cache needing to be wiped out are builds failing because they depend on a masked slot/subslot combination (and emerge asking you to remove a mask.) In this case, you will want to recursively remove the contents of {{f|/var/tmp/metro/cache/package-cache}}. For periodic updates of a consistent set of kits, this is typically not necessary.}}

Latest revision as of 03:09, April 15, 2024

Overview

For our local development setup, we will be using gitolite. Gitolite will make things quite a bit easier by managing git repositories for us. Think of gitolite as your own private GitHub that has no Web user interface (we modify its settings by pushing to its special gitolite-admin repo) and you'll have a pretty good idea of what gitolite does. To keep things simple, we will be using a single development system in our setup example below. However, because gitolite can be accessed over the network, you can easily set up gitolite on another system on your LAN or a remote system on the Internet and securely access it simply by modifying the git URL.

   Important

This document assumes you have basic knowledge of ssh-keygen and how to generate public/private SSH key pairs. If you don't know how to to this, see Generating SSH Keys for quick steps or OpenSSH Key Management, Part 1 for a more detailed introduction. For this article, you'll probably want to generate a private keys without a passphrase, which is more convenient but a much greater security risk if the private key gets compromised, or one with a passphrase but using keychain to manage ssh-agent for you.

Gitolite

Installation

To set up gitolite, we will use a repos user on your local development system. On this system, perform the following steps as root: 

repohost # useradd -m repos

The repos user will be a dedicated user account on the system that will have gitolite enabled and will house our git repositories. Now, we are going to su to this new user and perform gitolite configuration: 

repohost # su --login repos
user $ git clone https://github.com/sitaramc/gitolite
user $ install -d ~/bin

Now, as the repos user, add the following within your ~/.bashrc file:

   ~/.bashrc 
export PATH=$HOME/bin:$PATH

What we're doing is setting up a bin directory where the gitolite command will be installed, which will be in your path, so that you can use it more easily. With this done, perform the following steps: 

user $ source ~/.bashrc
user $ gitolite/install -ln

Now, your repos account is almost ready to be used for hosting repositories. The way gitolite works is that it is going to basically take over ssh access to the account, so that when you connect via ssh with git, it will perform its own authentication. For this to work, you will need to enable your own "master key" to access gitolite.

To do this, you'll want to decide from which account you'll want to administer gitolite itself. I prefer to use my "drobbins" account on local system, so I will copy my ssh public key from ~/.ssh/id_ed25519.pub to /var/tmp/drobbins.pub, and then perform the following steps to "prime" gitolite with this admin public key -- do this as the repos user: 

user $ gitolite setup -pk /var/tmp/drobbins.pub

Gitolite will now be initialized to recognize the drobbins account as an administrator, which will allow this account to clone from repos@localhost:gitolite-admin and push any changes to this special git repository which contains the master configuration for gitolite. This is important because we will be performing the rest of gitolite setup over ssh, using this account.

   Note

OK, gitolite is installed! From this point forward, we will be using the drobbins (or equivalent) account on your development workstation to configure gitolite.

   Note

If you are setting up gitolite on a separate server, and assuming you have ssh properly configured, it would be prudent at this point to test the connection to the remote server from your local workstation . Perform ssh -T, or ssh test command appropriate to your setup, as shown below to verify you can connect: 

user $ ssh -T repos@remote-server-name

You should receive confirmation message similar to: 

hello repos, this is repos@remote-server-name running gitolite3 v3.6.11-3-g39293e6 on git 2.19.1

 R W	gitolite-admin
 R W	testing

If you receive the following message during testing: 

perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
	LANGUAGE = (unset),
...

then localization settings for the remote server have not been completed. If remote server is Funtoo-based, complete the steps on the page Funtoo Linux Localization on the remote server.

gitolite-admin Clone

Now that gitolite is ready under the repos user on your local system, we can configure the rest by cloning the gitolite-admin repo and commiting configuration changes to it. I am going to use the drobbins account on the same system to do this, and you will use whatever account is associated with the public key you loaded into gitolite. I like storing my development repos in ~/development , so I'll go ahead and clone the gitolite-admin repo to that location so it can live along all my other git repos. Feel free to put this git repo wherever you like to store git repos that you develop on: 

user $ cd ~/development
user $ git clone repos@localhost:gitolite-admin
user $ cd gitolite-admin

We are now ready to configure gitolite. We'll do this by modifying conf/gitolite.conf in the git repo and adding new ssh public keys to keydir/ as needed. You will see that the initial public key you used to "prime" gitolite already exists in keydir/. Once we change the configuration, and potentially add new public ssh keys that we want to grant access to gitolite-managed repositories, we'll perform a git commit and git push, and if gitolite doesn't complain about our changes, they'll take effect immediately. We'll go through our initial configuration steps below.

gitolite Configuration

Since I will be generating meta-repo and kits using my drobbins user, this account will need to have permissions to create repositories in gitolite. We're ready to edit conf/gitolite.conf so that it looks like this:

   gitolite.conf 
# Group definitions below, starting with @. This makes it easy to associate multiple ssh keys with a particular person.

# Admins will have full control over gitolite.
@admins = drobbins

# Developers will be able to create, read and write git repositories.
@developers = drobbins

# Users will have read-only access.
@users = drobbins

# repositories:

# SPECIAL ADMIN REPO BELOW -- modify with care! 

repo gitolite-admin
    RW+     =   @admins

# AUTO-CREATED (wild) REPOS: gitolite will auto-create repos under wildrepo/ for us
# upon initial clone of any path within, if the repo doesn't already exist.

repo wildrepo/..*
    C       =   @developers
    RW+     =   @developers
    R       =   @developers @users

Now, we will want to commit and push our changes in the gitolite repo so they take effect -- This is how gitolite configuration changes are applied: 

user $ git add .
user $ git commit -a -m "Initial setup"
user $ git push
Enumerating objects: 10, done.
Counting objects: 100% (10/10), done.
Delta compression using up to 12 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 607 bytes

Configuration

Now, as your own user account, on your development system (which could be the same Funtoo system), you will want to emerge metatools or use git to clone the master branch of metatools. Here's how we would use the master branch: 

user $ emerge --onlydeps metatools
user $ mkdir development
user $ cd development
user $ git clone https://code.funtoo.org/bitbucket/scm/~drobbins/funtoo-metatools.git

If using the git master version, you will need to add the following to your ~/.bashrc :

    
export PYTHONPATH=~/development/funtoo-metatools
export PATH=~/development/funtoo-metatools:$PATH

Create Your Own Kit-Fixups

You will now want to clone kit-fixups from code.funtoo.org, and create a commit that updates ~/kit-fixups/releases/next/repositories.yaml to adjust the remotes as follows:

   releases/next/repositories.yaml (yaml source code) 
# Have all of our remote configurations defined in one place for ease of maintenance
  remotes:
    dev:
      url: repos@gitolitehost:wildrepo/staging/{repo}
    prod:
      url: repos@gitolitehost:wildrepo/staging/{repo}

Push this commit up to your personal kit-fixups.

Generating New Kits

With this all configured, you are ready to generate new kits. These kits will be generated as root on your development system, and will be pushed up to repos@localhost/staging/repo_name. The next is the release of Funtoo. 

user $ merge-kits next --prod --fixups_url=ssh://git@code.funtoo.org:7999/myusername/kit-fixups.git --create_branches


merge-kits will proceed to create new kits and meta-repo, and will push them up to repos@localhost:wildrepo/staging/meta-repo, repos@localhost:wildrepo/staging/core-kit, etc. This process can take quite a while but has been optimized to run quickly on multi-core systems and use caching extensively so successive runs will complete quickly.

Using New Kits

Now that the new meta-repo and kits are created, here's how you'll use them on an existing Funtoo system, instead of your official Funtoo meta-repo and kits. First, we'll want to modify /etc/ego.conf as follows:

   /etc/ego.conf 
[global]

sync_user = drobbins
sync_base_url = repos@gitolitehost:wildrepo/staging/{repo}

# Yes, you are supposed to have a literal "{repo}", above. Ego recognizes this special pattern.

# You can have whatever [kits] section you want, below...

Since we are syncing as a regular user, we will need to make sure that our user account is in the portage group and that /var/git is writable by, in this case, drobbins: 

root # chmod g+rwx /var/git

Let's also move the current meta-repo out of the way -- you can also simply delete the existing meta-repo. And then we'll re-run ego sync: 

root # cd /var/git
root # mv meta-repo meta-repo.official
root # exit
user $ ego sync
Running as regular user.
Syncing meta-repo
Cloning into '/var/git/meta-repo'...
X11 forwarding request failed on channel 0
remote: Enumerating objects: 13, done.
remote: Counting objects: 100% (13/13), done.
remote: Compressing objects: 100% (11/11), done.
remote: Total 13 (delta 1), reused 0 (delta 0), pack-reused 0
Receiving objects: 100% (13/13), done.
Resolving deltas: 100% (1/1), done.
Syncing browser-kit branch 1.4-release

If you type emerge -auDN @world, ego will now be using your custom kits, rather than the official Funtoo ones. This means that you can perform a variety of things you couldn't before. You can now add your own custom ebuilds to your fork of kit-fixups, and merge-kits will automatically incorporate these changes into your own custom kits. This will allow you to locally test any changes before submitting them as pull requests to Funtoo. You will also be able to maintain your own meta-repo and kits with your own local modifications, and have your systems use these meta-repo/kits instead of the official Funtoo ones.

Using New Kits with Metro

Rather than using new kits on a local Funtoo Linux system, you may want to use the kits to perform a metro build. This is desirable when you may want to use metro to perform continuous integration by attempting to perform a full stage1, 2 and 3 build, which is what the steps in this example accomplish, or simply test some hypothetical changes to the tree before creating a PR. Let's look at how to do this.

First, since metro runs as root, you will need to ensure that the root user on your local system has read access to gitolite. Perform the following steps: 

root # cp /root/.ssh/id_ed25519.pub /var/tmp/root.pub
root # exit
user $ cd ~/development/gitolite-admin/keydir
user $ cp /var/tmp/root.pub .
user $ git add *

Now, give the root.pub key the ability to have read-only access to the repository (showing only the changes required):

   ~/development/gitolite-admin/conf/gitolite.conf 
@users = drobbins root

Now, commit changes: 

user $ git add .
user $ git commit -a -m "gitolite config updates"

Now, the root user, which is used by Metro to perform the ego sync, should be able to connect to the repository.

Next, we will want to instruct Metro to grab the custom meta-repo instead of the official one. To do this, first set up Metro on your system. Then, prior to running metro, set the EGO_SYNC_BASE_URL in your environment as follows: 

root # export EGO_SYNC_BASE_URL="repos@localhost:wildrepo/staging/{repo}"

Metro should grab these settings and automatically adjust the ego.conf it uses based on these settings, to grab your meta-repo instead of the official one. You should now be able to perform local or CI testing of your changes. It's always a good idea to put this setting in ~/.bashrc of a CI user account so that it is always set correctly.

Retrieved from "https://www.funtoo.org/index.php?title=Funtoo:Metatools/Advanced_Usage/Gitolite_Setup&oldid=43592"