Difference between revisions of "Funtoo:Metro/Manual Setup"

Revision as of 21:19, February 20, 2019

This page documents how to set up your metro build repository manually. Most people will be better served by following the auto-setup instructions at Metro/AutoSetup. However, advanced users will glean a lot of useful information regarding how Metro manages its build repository from this document. In addition, the second section of this document will cover the concept of remote builds -- using a more "generic" stage3 as a seed stage to bootstrap the build of a more optimized stage3.

Introduction

In the following example we are creating a pentium4 stage 3 compiled for x86-32bit binary compatibility. Pentium4 is a subarch of the x86-32bit architecture. Once you have metro installed you may find a full list of each subarch in your /var/git/meta-repo/kits/core-kit/profiles/funtoo-1.0/linux-gnu/arch/x86-32bit/subarch directory: Example:

root # ls /var/git/meta-repo/kits/core-kit/profiles/funtoo/1.0/linux-gnu/arch/x86-32bit/subarch/
amd64-k8+sse3_32  athlon-4      athlon-xp  core2_32    i486  k6-2       pentium      pentium2  pentiumpro
amd64-k8_32       athlon-mp     atom_32    generic_32  i686  k6-3       pentium-m    pentium3  prescott
athlon            athlon-tbird  btver1     geode       k6    native_32  pentium-mmx  pentium4  xen-pentium4+sse3

64-bit PC profiles can be found in the /var/git/meta-repo/kits/core-kit/profiles/funtoo/1.0/linux-gnu/arch/x86-64bit/subarch/ directory:

root # ls /var/git/meta-repo/kits/core-kit/profiles/funtoo/1.0/linux-gnu/arch/x86-64bit/subarch/
amd64-bulldozer  amd64-k8+sse3      btver1_64   generic_64         intel64-nehalem      native_64
amd64-jaguar     amd64-piledriver   core-avx-i  intel64-broadwell  intel64-sandybridge  nocona
amd64-k10        amd64-steamroller  core2_64    intel64-haswell    intel64-silvermont   opteron_64
amd64-k8         atom_64            corei7      intel64-ivybridge  intel64-westmere     xen-pentium4+sse3_64

First stages build (local build)

To get this all started, we need to bootstrap the process by downloading an initial seed stage3 to use for building and place it in its proper location in /home/mirror/funtoo, so that Metro can find it. We will also need to create some special "control" files in /home/mirror/funtoo, which will allow Metro to understand how it is supposed to proceed.

Step 1: Set up pentium4 repository (local build)

Assuming we're following the basic steps outlined in the previous section, and building 1.3-release-std build for the pentium4, using a generic pentium4 stage3 as a seed stage, then here the first set of steps we'd perform:

root # install -d /home/mirror/funtoo/1.3-release-std/x86-32bit/pentium4
root # install -d /home/mirror/funtoo/1.3-release-std/snapshots
root # cd /home/mirror/funtoo/1.3-release-std/x86-32bit/pentium4
root # install -d 2019-02-09
root # cd 2019-02-09
root # wget -c https://build.funtoo.org/1.3-release-std/x86-32bit/pentium4/2019-02-09/stage3-pentium4-1.3-release-std-2019-02-09.tar.xz
root # cd ..
root # install -d .control/version
root # echo "2019-02-09" > .control/version/stage3
root # install -d .control/strategy
root # echo local >  .control/strategy/build
root # echo stage3 > .control/strategy/seed

OK, let's review the steps above. First, we create the directory /home/mirror/funtoo/1.3-release-std/x86-32bit/pentium4, which is where Metro will expect to find 1.3-release-std pentium4 builds -- it is configured to look here by default. Then we create a specially-named directory to house our seed x86 stage3. Again, by default, Metro expects the directory to be named this way. We enter this directory, and download our seed x86 stage3 from funtoo.org. Note that the 2019-02-09 version stamp matches. Make sure that your directory name matches the stage3 name too. Everything has been set up to match Metro's default filesystem layout.

Next, we go back to the /home/mirror/metro/1.3-release-std/x86-32bit/pentium4 directory, and inside it, we create a .control directory. This directory and its subdirectories contain special files that Metro references to determine certain aspects of its behavior. The .control/version/stage3 file is used by Metro to track the most recently-built stage3 for this particular build and subarch. Metro will automatically update this file with a new version stamp after it successfully builds a new stage3. But because Metro didn't actually build this stage3, we need to set up the .control/version/stage3 file manually. This will allow Metro to find our downloaded stage3 when we set up our pentium4 build to use it as a seed. Also note that Metro will create a similar .control/version/stage1 file after it successfully builds an pentium4 1.3-release-std stage1.

We also set up .control/strategy/build and .control/strategy/seed files with values of local and stage3 respectively. These files define the building strategy Metro will use when we build pentium4 1.3-release-std stages. With a build strategy of local, Metro will source its seed stage from 1.3-release-std pentium4, the current directory. And with a seed strategy of stage3, Metro will use a stage3 as a seed, and use this seed to build a new stage1, stage2 and stage3.

Step 2: Building the pentium4 stages

Incidentally, if all you wanted to do at this point was to build a new pentium4 1.3-release-std stage1/2/3 (plus openvz and vserver templates). You would begin the process by typing:

root # cd /root/metro
root # scripts/ezbuild.sh 1.3-release-std x86-32bit pentium4

If you have a slow machine, it could take several hours to be completed because several "heavy" components like gcc or glibc have to be recompiled in each stage. Once a stage has been successfully completed, it is placed in the "${METRO_MIRROR}/1.3-release-std/x32-bit/pentium4/YYYY-MM-DD" subdirectory, where YYYY-MM-DD is today's date at the time the ezbuild.sh script was started or the date you put on the ezscript.sh command line.

Building for another binary compatible architecture (remote build)

As written above, Metro is able to perform what is called a remote build --- building different architecture stage3 from a binary compatible seed stage3 (e.g. using a pentium4 stage3 to seed a Intel Core2 32bit stage3).

In the Metro terminology this is called a remote build -- a stage 3 of a different, but binary compatible, architecture is used as a seed. It is called a "remote build" because it is you can use a build, subarch or even arch that is not the one you are building for. As long as the CPU instruction set of the remote build is a subset of the build you are trying to create, it will work fine. For example, you can use a generic 64-bit build (arch_desc of x86-64bit, subarch) to build any PC-compatible 64-bit stages that have a more feature-rich x86-64bit instruction set (such as intel64-sandybridge, for example.)

Note that we are not talking about using the magic of the QEMU emulator to build a typically totally binary incompatible stage3 -- like an arm-32bit stage on an x86-64bit system. For information on how to do that, see the QEMU instructions on the main Metro page.

The best way to understand the nomenclature of "remote" build is that your build will be using a seed stage from "outside" its own repository path of /home/mirror/funtoo/(build)/(arch_desc)/(subarch) -- thus, the seed stage is "remote".

Important

Please note that Metro's concept of "remote" build may be different from what you assume -- read the definition above very carefully to make sure you grasp it!

Step 1: Set up Core_2 32bit repository

In this example, we're going to use this pentium4 1.3-release-std stage3 to seed a new Core_2 32bit 1.3-release-std build. To get that done, we need to set up the pentium4 build directory as follows:

root #  cd /home/mirror/funtoo/1.3-release-std/x86-32bit
root # install -d core2_32
root # cd core2_32
root # install -d .control/strategy
root # echo remote > .control/strategy/build
root # echo stage3 > .control/strategy/seed
root # install -d .control/remote
root # echo 1.3-release-std > .control/remote/build
root # echo x86-32bit > .control/remote/arch_desc
root # echo pentium4 > .control/remote/subarch

The steps we follow are similar to those we performed for a local build to set up our pentium4 directory for local build. However, note the differences. We didn't download a stage, because we are going to use the pentium4 stage to build a new Core_2 32bit stage. We also didn't create the .control/version/stage{1,3} files because Metro will create them for us after it successfully builds a new stage1 and stage3. We are still using a stage3 seed strategy, but we've set the build strategy to remote, which means that we're going to use a seed stage that's not from this particular subdirectory. Where are we going to get it from? The .control/remote directory contains this information, and lets Metro know that it should look for its seed stage3 in the /home/mirror/funtoo/1.3-release-std/x86-32bit/pentium4 directory. Which one will it grab? You guessed it -- the most recently built stage3 (since our seed strategy was set to stage3) that has the version stamp of 2018-12-24, as recorded in /home/mirror/1.3-release-std/x86-32bit/pentium4/.control/version/stage. Now you can see how all those control files come together to direct Metro to do the right thing.

Note

arch_desc should be set to one of: x86-32bit, x86-64bit or pure64 for PC-compatible systems. You must use a 32-bit build as a seed for other 32-bit builds, and a 64-bit build as a seed for other 64-bit builds.

Step 2: Building the Core_2 32bit stages

Now, you could start building your new Core_2 32bit stage1/2/3 by typing the following:

root # /root/metro/scripts/ezbuild.sh 1.3-release-std x86-32bit core2_32 full

In this case, the produced stages are placed in the /home/mirror/funtoo/1.3-release-std/x32-bit/core2_32/YYYY-MM-DD subdirectory.

Step 3: The Next Build

At this point, you now have a new Core_2 32bit stage3, built using a "remote" pentium4 stage3. Once the first remote build completes successfully, metro will automatically change .control/strategy/build to be local instead of remote, so it will use the most recently-built Core_2 32bit stage3 as a seed for any new Core_2 32bit builds from now on.

@@ Line 25: / Line 25: @@
 To get this all started, we need to bootstrap the process by downloading an initial seed stage3 to use for building and place it in its proper location in {{f|/home/mirror/funtoo}}, so that Metro can find it. We will also need to create some special &quot;control&quot; files in {{f|/home/mirror/funtoo}}, which will allow Metro to understand how it is supposed to proceed.
-== Step 1: Set up pentium4 repository (local build) == <!--T:45-->
+=== Step 1: Set up pentium4 repository (local build) === <!--T:45-->
 <!--T:46-->
@@ Line 52: / Line 52: @@
 We also set up {{f|.control/strategy/build}} and {{f|.control/strategy/seed}} files with values of {{f|local}} and {{f|stage3}} respectively. These files define the building strategy Metro will use when we build pentium4 1.3-release-std stages. With a build strategy of {{f|local}}, Metro will source its seed stage from 1.3-release-std pentium4, the current directory. And with a seed strategy of {{f|stage3}}, Metro will use a stage3 as a seed, and use this seed to build a new stage1, stage2 and stage3.
-== Step 2: Building the pentium4 stages == <!--T:51-->
+=== Step 2: Building the pentium4 stages === <!--T:51-->
 Incidentally, if all you wanted to do at this point was to build a new pentium4 1.3-release-std stage1/2/3 (plus openvz and vserver templates). You would begin the process by typing: