The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.
Difference between revisions of "Funtoo:Metro/Manual Setup"
| (7 intermediate revisions by 3 users not shown) | |||
| Line 1: | Line 1: | ||
This page documents how to set up your metro build repository manually. | This page documents how to set up your metro build repository manually. Most people will be better served by following the auto-setup instructions at [[Funtoo:Metro/AutoSetup|AutoSetup]]. However, advanced users will glean a lot of useful information regarding how Metro manages its build repository from this document. In addition, the second section of this document will cover the concept of ''remote builds'' -- using a more "generic" stage3 as a seed stage to bootstrap the build of a more optimized stage3. | ||
== Introduction == | |||
In the following example we are creating a pentium4 stage 3 compiled for x86-32bit binary compatibility. Pentium4 is a subarch of the x86-32bit architecture. Once you have metro installed you may find a full list of each subarch in your {{f|/var/git/meta-repo/kits/core-kit/profiles/funtoo-1.0/linux-gnu/arch/x86-32bit/subarch}} directory: | In the following example we are creating a pentium4 stage 3 compiled for x86-32bit binary compatibility. Pentium4 is a subarch of the x86-32bit architecture. Once you have metro installed you may find a full list of each subarch in your {{f|/var/git/meta-repo/kits/core-kit/profiles/funtoo-1.0/linux-gnu/arch/x86-32bit/subarch}} directory: | ||
Example: | Example: | ||
| Line 20: | Line 21: | ||
}} | }} | ||
= First stages build (local build) = | == First stages build (local build) == | ||
To get this all started, we need to bootstrap the process by downloading an initial seed stage3 to use for building and place it in its proper location in {{f|/home/mirror/funtoo}}, so that Metro can find it. We will also need to create some special "control" files in {{f|/home/mirror/funtoo}}, which will allow Metro to understand how it is supposed to proceed. | To get this all started, we need to bootstrap the process by downloading an initial seed stage3 to use for building and place it in its proper location in {{f|/home/mirror/funtoo}}, so that Metro can find it. We will also need to create some special "control" files in {{f|/home/mirror/funtoo}}, which will allow Metro to understand how it is supposed to proceed. | ||
== Step 1: Set up pentium4 repository (local build) == <!--T:45--> | === Step 1: Set up pentium4 repository (local build) === <!--T:45--> | ||
<!--T:46--> | <!--T:46--> | ||
Assuming we're following the basic steps outlined in the previous section, and building {{f| | Assuming we're following the basic steps outlined in the previous section, and building {{f|1.4-release-std}} build for the {{f|pentium4}}, using a generic {{f|pentium4}} stage3 as a seed stage, then here the first set of steps we'd perform: | ||
<!--T:47--> | <!--T:47--> | ||
{{console|body= | {{console|body= | ||
# ##i##install -d /home/mirror/funtoo/ | # ##i##install -d /home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4 | ||
# ##i##install -d /home/mirror/funtoo/ | # ##i##install -d /home/mirror/funtoo/1.4-release-std/snapshots | ||
# ##i##cd /home/mirror/funtoo/ | # ##i##cd /home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4 | ||
# ##i##install -d | # ##i##install -d 2019-02-09 | ||
# ##i##cd | # ##i##cd 2019-02-09 | ||
# ##i##wget -c https://build.funtoo.org/ | # ##i##wget -c https://build.funtoo.org/1.4-release-std/x86-32bit/pentium4/2019-02-09/stage3-pentium4-1.4-release-std-2019-02-09.tar.xz | ||
# ##i##cd .. | # ##i##cd .. | ||
# ##i##install -d .control/version | # ##i##install -d .control/version | ||
# ##i##echo " | # ##i##echo "2019-02-09" > .control/version/stage3 | ||
# ##i##install -d .control/strategy | # ##i##install -d .control/strategy | ||
# ##i##echo local > .control/strategy/build | # ##i##echo local > .control/strategy/build | ||
| Line 46: | Line 46: | ||
}} | }} | ||
OK, let's review the steps above. First, we create the directory {{f|/home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4}}, which is where Metro will expect to find {{f|1.4-release-std}} pentium4 builds -- it is configured to look here by default. Then we create a specially-named directory to house our seed x86 stage3. Again, by default, Metro expects the directory to be named this way. We enter this directory, and download our seed x86 stage3 from funtoo.org. Note that the {{f|2019-02-09}} version stamp matches. Make sure that your directory name matches the stage3 name too. Everything has been set up to match Metro's default filesystem layout. | |||
OK, let's review the steps above. First, we create the directory {{f|/home/mirror/funtoo/ | |||
Next, we go back to the {{f|/home/mirror/metro/1.4-release-std/x86-32bit/pentium4}} directory, and inside it, we create a {{f|.control}} directory. This directory and its subdirectories contain special files that Metro references to determine certain aspects of its behavior. The {{f|.control/version/stage3}} file is used by Metro to track the most recently-built stage3 for this particular build and subarch. Metro will automatically update this file with a new version stamp after it successfully builds a new stage3. But because Metro didn't actually ''build'' this stage3, we need to set up the {{f|.control/version/stage3}} file manually. This will allow Metro to find our downloaded stage3 when we set up our pentium4 build to use it as a seed. Also note that Metro will create a similar {{f|.control/version/stage1}} file after it successfully builds an pentium4 1.4-release-std stage1. | |||
Next, we go back to the {{f|/home/mirror/metro/ | |||
We also set up {{f|.control/strategy/build}} and {{f|.control/strategy/seed}} files with values of {{f|local}} and {{f|stage3}} respectively. These files define the building strategy Metro will use when we build pentium4 1.4-release-std stages. With a build strategy of {{f|local}}, Metro will source its seed stage from 1.4-release-std pentium4, the current directory. And with a seed strategy of {{f|stage3}}, Metro will use a stage3 as a seed, and use this seed to build a new stage1, stage2 and stage3. | |||
We also set up {{f|.control/strategy/build}} and {{f|.control/strategy/seed}} files with values of {{f|local}} and {{f|stage3}} respectively. These files define the building strategy Metro will use when we build pentium4 | |||
== Step 2: Building the pentium4 stages == <!--T:51--> | === Step 2: Building the pentium4 stages === <!--T:51--> | ||
Incidentally, if all you wanted to do at this point was to build a new pentium4 1.4-release-std stage1/2/3 (plus openvz and vserver templates). You would begin the process by typing: | |||
Incidentally, if all you wanted to do at this point was to build a new pentium4 | |||
{{console|body= | {{console|body= | ||
# ##i##cd /root/metro | # ##i##cd /root/metro | ||
# ##i##scripts/ezbuild.sh | # ##i##scripts/ezbuild.sh 1.4-release-std x86-32bit pentium4 | ||
}} | }} | ||
If you have a slow machine, it could take several hours to be completed because several "heavy" components like gcc or glibc have to be recompiled in each stage. Once a stage has been successfully completed, it is placed in the {{f|"${METRO_MIRROR}/1.4-release-std/x32-bit/pentium4/YYYY-MM-DD"}} subdirectory, where {{f|YYYY-MM-DD}} is today's date at the time the {{f|ezbuild.sh}} script was started or the date you put on the ezscript.sh command line. | |||
If you have a slow machine, it could take several hours to be completed because several "heavy" components like gcc or glibc have to be recompiled in each stage. Once a stage has been successfully completed, it is placed in the {{f|"${METRO_MIRROR}/ | |||
== Building for another binary compatible architecture (remote build) == | |||
As written above, Metro is able to perform what is called a '''remote build''' --- building different architecture stage3 from a binary compatible seed stage3 (e.g. using a pentium4 stage3 to seed a {{c|Intel Core2 32bit}} stage3). | |||
In the Metro terminology this is called a '''remote build''' -- a stage 3 of a different, but binary compatible, architecture is used as a seed. It is called a "remote build" because it is you can use a build, subarch or even arch that is not the one you are building for. As long as the CPU instruction set of the remote build is a subset of the build you are trying to create, it will work fine. For example, you can use a generic 64-bit build ({{c|arch_desc}} of {{c|x86-64bit}}, {{c|subarch|generic_64}}) to build any PC-compatible 64-bit stages that have a more feature-rich x86-64bit instruction set (such as {{c|intel64-sandybridge}}, for example.) | |||
Note that we are not talking about using the magic of the QEMU emulator to build a typically totally binary incompatible stage3 -- like an {{c|arm-32bit}} stage on an x86-64bit system. For information on how to do that, see the QEMU instructions on [[Metro|the main Metro page.]] | |||
The best way to understand the nomenclature of "remote" build is that your build will be using a seed stage from "outside" its own repository path of {{f|/home/mirror/funtoo/(build)/(arch_desc)/(subarch)}} -- thus, the seed stage is "remote". | |||
{{Important|Please note that Metro's concept of "remote" build may be different from what you assume -- read the definition above very carefully to make sure you grasp it!}} | |||
{{ | |||
== Step 1: Set up Core_2 32bit repository == | === Step 1: Set up Core_2 32bit repository === | ||
In this example, we're going to use this pentium4 1.4-release-std stage3 to seed a new Core_2 32bit 1.4-release-std build. To get that done, we need to set up the pentium4 build directory as follows: | |||
In this example, we're going to use this pentium4 | |||
{{console|body= | {{console|body= | ||
# ##i## cd /home/mirror/funtoo/ | # ##i## cd /home/mirror/funtoo/1.4-release-std/x86-32bit | ||
# ##i##install -d core2_32 | # ##i##install -d core2_32 | ||
# ##i##cd core2_32 | # ##i##cd core2_32 | ||
| Line 95: | Line 87: | ||
# ##i##echo stage3 > .control/strategy/seed | # ##i##echo stage3 > .control/strategy/seed | ||
# ##i##install -d .control/remote | # ##i##install -d .control/remote | ||
# ##i##echo | # ##i##echo 1.4-release-std > .control/remote/build | ||
# ##i##echo x86-32bit > .control/remote/arch_desc | # ##i##echo x86-32bit > .control/remote/arch_desc | ||
# ##i##echo pentium4 > .control/remote/subarch | # ##i##echo pentium4 > .control/remote/subarch | ||
}} | }} | ||
The steps we follow are similar to those we performed for a ''local build'' to set up our pentium4 directory for local build. However, note the differences. We didn't download a stage, because we are going to use the pentium4 stage to build a new Core_2 32bit stage. We also didn't create the {{c|.control/version/stage{1,3}}} files because Metro will create them for us after it successfully builds a new stage1 and stage3. We are still using a {{c|stage3}} seed strategy, but we've set the build strategy to {{c|remote}}, which means that we're going to use a seed stage that's not from this particular subdirectory. Where are we going to get it from? The {{c|.control/remote}} directory contains this information, and lets Metro know that it should look for its seed stage3 in the {{c|/home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4}} directory. Which one will it grab? You guessed it -- the most recently built ''stage3'' (since our seed strategy was set to {{c|stage3}}) that has the version stamp of {{c|2018-12-24}}, as recorded in {{c|/home/mirror/1.4-release-std/x86-32bit/pentium4/.control/version/stage}}. Now you can see how all those control files come together to direct Metro to do the right thing. | |||
The steps we follow are similar to those we performed for a ''local build'' to set up our pentium4 directory for local build. However, note the differences. We didn't download a stage, because we are going to use the pentium4 stage to build a new Core_2 32bit stage. We also didn't create the | |||
{{Note|{{c|arch_desc}} should be set to one of: {{c|x86-32bit}}, {{c|x86-64bit}} or {{c|pure64}} for PC-compatible systems. You must use a 32-bit build as a seed for other 32-bit builds, and a 64-bit build as a seed for other 64-bit builds.}} | |||
{{Note| | |||
== Step 2: Building the Core_2 32bit stages == | === Step 2: Building the Core_2 32bit stages === | ||
Now, you could start building your new Core_2 32bit stage1/2/3 by typing the following: | |||
Now, you could start building your new Core_2 32bit stage1/2/3 | |||
{{console|body= | {{console|body= | ||
# ##i##/root/metro/scripts/ezbuild.sh | # ##i##/root/metro/scripts/ezbuild.sh 1.4-release-std x86-32bit core2_32 full | ||
}} | }} | ||
In this case, the produced stages are placed in the {{c|/home/mirror/funtoo/1.4-release-std/x32-bit/core2_32/YYYY-MM-DD}} subdirectory. | |||
In | |||
== Step 3: The Next Build == | === Step 3: The Next Build === | ||
At this point, you now have a new Core_2 32bit stage3, built using a "remote" pentium4 stage3. Once the first remote build completes successfully, metro will automatically change {{c|.control/strategy/build}} to be {{c|local}} instead of {{c|remote}}, so it will use the most recently-built Core_2 32bit stage3 as a seed for any new Core_2 32bit builds from now on. | At this point, you now have a new Core_2 32bit stage3, built using a "remote" pentium4 stage3. Once the first remote build completes successfully, metro will automatically change {{c|.control/strategy/build}} to be {{c|local}} instead of {{c|remote}}, so it will use the most recently-built Core_2 32bit stage3 as a seed for any new Core_2 32bit builds from now on. | ||
[[Category:Official Documentation]] | [[Category:Official Documentation]] | ||
[[Category:Metro]] | [[Category:Metro]] | ||
Latest revision as of 22:24, October 10, 2022
This page documents how to set up your metro build repository manually. Most people will be better served by following the auto-setup instructions at AutoSetup. However, advanced users will glean a lot of useful information regarding how Metro manages its build repository from this document. In addition, the second section of this document will cover the concept of remote builds -- using a more "generic" stage3 as a seed stage to bootstrap the build of a more optimized stage3.
Introduction
In the following example we are creating a pentium4 stage 3 compiled for x86-32bit binary compatibility. Pentium4 is a subarch of the x86-32bit architecture. Once you have metro installed you may find a full list of each subarch in your /var/git/meta-repo/kits/core-kit/profiles/funtoo-1.0/linux-gnu/arch/x86-32bit/subarch directory:
Example:
root # ls /var/git/meta-repo/kits/core-kit/profiles/funtoo/1.0/linux-gnu/arch/x86-32bit/subarch/ amd64-k8+sse3_32 athlon-4 athlon-xp core2_32 i486 k6-2 pentium pentium2 pentiumpro amd64-k8_32 athlon-mp atom_32 generic_32 i686 k6-3 pentium-m pentium3 prescott athlon athlon-tbird btver1 geode k6 native_32 pentium-mmx pentium4 xen-pentium4+sse3
64-bit PC profiles can be found in the /var/git/meta-repo/kits/core-kit/profiles/funtoo/1.0/linux-gnu/arch/x86-64bit/subarch/ directory:
root # ls /var/git/meta-repo/kits/core-kit/profiles/funtoo/1.0/linux-gnu/arch/x86-64bit/subarch/ amd64-bulldozer amd64-k8+sse3 btver1_64 generic_64 intel64-nehalem native_64 amd64-jaguar amd64-piledriver core-avx-i intel64-broadwell intel64-sandybridge nocona amd64-k10 amd64-steamroller core2_64 intel64-haswell intel64-silvermont opteron_64 amd64-k8 atom_64 corei7 intel64-ivybridge intel64-westmere xen-pentium4+sse3_64
First stages build (local build)
To get this all started, we need to bootstrap the process by downloading an initial seed stage3 to use for building and place it in its proper location in /home/mirror/funtoo, so that Metro can find it. We will also need to create some special "control" files in /home/mirror/funtoo, which will allow Metro to understand how it is supposed to proceed.
Step 1: Set up pentium4 repository (local build)
Assuming we're following the basic steps outlined in the previous section, and building 1.4-release-std build for the pentium4, using a generic pentium4 stage3 as a seed stage, then here the first set of steps we'd perform:
root # install -d /home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4 root # install -d /home/mirror/funtoo/1.4-release-std/snapshots root # cd /home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4 root # install -d 2019-02-09 root # cd 2019-02-09 root # wget -c https://build.funtoo.org/1.4-release-std/x86-32bit/pentium4/2019-02-09/stage3-pentium4-1.4-release-std-2019-02-09.tar.xz root # cd .. root # install -d .control/version root # echo "2019-02-09" > .control/version/stage3 root # install -d .control/strategy root # echo local > .control/strategy/build root # echo stage3 > .control/strategy/seed
OK, let's review the steps above. First, we create the directory /home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4, which is where Metro will expect to find 1.4-release-std pentium4 builds -- it is configured to look here by default. Then we create a specially-named directory to house our seed x86 stage3. Again, by default, Metro expects the directory to be named this way. We enter this directory, and download our seed x86 stage3 from funtoo.org. Note that the 2019-02-09 version stamp matches. Make sure that your directory name matches the stage3 name too. Everything has been set up to match Metro's default filesystem layout.
Next, we go back to the /home/mirror/metro/1.4-release-std/x86-32bit/pentium4 directory, and inside it, we create a .control directory. This directory and its subdirectories contain special files that Metro references to determine certain aspects of its behavior. The .control/version/stage3 file is used by Metro to track the most recently-built stage3 for this particular build and subarch. Metro will automatically update this file with a new version stamp after it successfully builds a new stage3. But because Metro didn't actually build this stage3, we need to set up the .control/version/stage3 file manually. This will allow Metro to find our downloaded stage3 when we set up our pentium4 build to use it as a seed. Also note that Metro will create a similar .control/version/stage1 file after it successfully builds an pentium4 1.4-release-std stage1.
We also set up .control/strategy/build and .control/strategy/seed files with values of local and stage3 respectively. These files define the building strategy Metro will use when we build pentium4 1.4-release-std stages. With a build strategy of local, Metro will source its seed stage from 1.4-release-std pentium4, the current directory. And with a seed strategy of stage3, Metro will use a stage3 as a seed, and use this seed to build a new stage1, stage2 and stage3.
Step 2: Building the pentium4 stages
Incidentally, if all you wanted to do at this point was to build a new pentium4 1.4-release-std stage1/2/3 (plus openvz and vserver templates). You would begin the process by typing:
root # cd /root/metro root # scripts/ezbuild.sh 1.4-release-std x86-32bit pentium4
If you have a slow machine, it could take several hours to be completed because several "heavy" components like gcc or glibc have to be recompiled in each stage. Once a stage has been successfully completed, it is placed in the "${METRO_MIRROR}/1.4-release-std/x32-bit/pentium4/YYYY-MM-DD" subdirectory, where YYYY-MM-DD is today's date at the time the ezbuild.sh script was started or the date you put on the ezscript.sh command line.
Building for another binary compatible architecture (remote build)
As written above, Metro is able to perform what is called a remote build --- building different architecture stage3 from a binary compatible seed stage3 (e.g. using a pentium4 stage3 to seed a Intel Core2 32bit stage3).
In the Metro terminology this is called a remote build -- a stage 3 of a different, but binary compatible, architecture is used as a seed. It is called a "remote build" because it is you can use a build, subarch or even arch that is not the one you are building for. As long as the CPU instruction set of the remote build is a subset of the build you are trying to create, it will work fine. For example, you can use a generic 64-bit build (arch_desc of x86-64bit, subarch) to build any PC-compatible 64-bit stages that have a more feature-rich x86-64bit instruction set (such as intel64-sandybridge, for example.)
Note that we are not talking about using the magic of the QEMU emulator to build a typically totally binary incompatible stage3 -- like an arm-32bit stage on an x86-64bit system. For information on how to do that, see the QEMU instructions on the main Metro page.
The best way to understand the nomenclature of "remote" build is that your build will be using a seed stage from "outside" its own repository path of /home/mirror/funtoo/(build)/(arch_desc)/(subarch) -- thus, the seed stage is "remote".
Please note that Metro's concept of "remote" build may be different from what you assume -- read the definition above very carefully to make sure you grasp it!
Step 1: Set up Core_2 32bit repository
In this example, we're going to use this pentium4 1.4-release-std stage3 to seed a new Core_2 32bit 1.4-release-std build. To get that done, we need to set up the pentium4 build directory as follows:
root # cd /home/mirror/funtoo/1.4-release-std/x86-32bit root # install -d core2_32 root # cd core2_32 root # install -d .control/strategy root # echo remote > .control/strategy/build root # echo stage3 > .control/strategy/seed root # install -d .control/remote root # echo 1.4-release-std > .control/remote/build root # echo x86-32bit > .control/remote/arch_desc root # echo pentium4 > .control/remote/subarch
The steps we follow are similar to those we performed for a local build to set up our pentium4 directory for local build. However, note the differences. We didn't download a stage, because we are going to use the pentium4 stage to build a new Core_2 32bit stage. We also didn't create the .control/version/stage{1,3} files because Metro will create them for us after it successfully builds a new stage1 and stage3. We are still using a stage3 seed strategy, but we've set the build strategy to remote, which means that we're going to use a seed stage that's not from this particular subdirectory. Where are we going to get it from? The .control/remote directory contains this information, and lets Metro know that it should look for its seed stage3 in the /home/mirror/funtoo/1.4-release-std/x86-32bit/pentium4 directory. Which one will it grab? You guessed it -- the most recently built stage3 (since our seed strategy was set to stage3) that has the version stamp of 2018-12-24, as recorded in /home/mirror/1.4-release-std/x86-32bit/pentium4/.control/version/stage. Now you can see how all those control files come together to direct Metro to do the right thing.
arch_desc should be set to one of: x86-32bit, x86-64bit or pure64 for PC-compatible systems. You must use a 32-bit build as a seed for other 32-bit builds, and a 64-bit build as a seed for other 64-bit builds.
Step 2: Building the Core_2 32bit stages
Now, you could start building your new Core_2 32bit stage1/2/3 by typing the following:
root # /root/metro/scripts/ezbuild.sh 1.4-release-std x86-32bit core2_32 full
In this case, the produced stages are placed in the /home/mirror/funtoo/1.4-release-std/x32-bit/core2_32/YYYY-MM-DD subdirectory.
Step 3: The Next Build
At this point, you now have a new Core_2 32bit stage3, built using a "remote" pentium4 stage3. Once the first remote build completes successfully, metro will automatically change .control/strategy/build to be local instead of remote, so it will use the most recently-built Core_2 32bit stage3 as a seed for any new Core_2 32bit builds from now on.