The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.
How Metro Works
Introduction
Metro is a somewhat complex set of shell (specifically bash) and python scripts and various files that automagically creates stages 1/2/3 tarballs used to emerge your Funtoo Linux system. Of course, Metro needs a seed to start its work and will recompile things like the so well known gcc and glibc. A 'seed' is either a pre-built stage3 archive available on any Funtoo Linux mirror or a snapshot of your current Funtoo Linux installation.
Arborescence description
Overview
Don't search through your bin directories, everything lies in /usr/lib/metro. An unfolded view as of Metro revision 1.5.1 is this one :
├── BUGS
├── ChangeLog
├── TODO
├── etc
│ ├── builds
│ │ ├── funtoo
│ │ │ ├── build.conf
│ │ │ └── multi-targets
│ │ │ ├── freshen
│ │ │ ├── full
│ │ │ ├── quick
│ │ │ ├── stage1
│ │ │ └── strategy
│ │ │ └── seed
│ │ │ ├── stage1
│ │ │ └── stage3
│ │ ├── gentoo
│ │ │ ├── build.conf
│ │ │ └── multi-targets
│ │ │ ├── freshen
│ │ │ ├── full
│ │ │ ├── quick
│ │ │ ├── stage1
│ │ │ └── strategy
│ │ │ └── seed
│ │ │ ├── stage1
│ │ │ └── stage3
│ │ ├── hollow
│ │ │ ├── build.conf
│ │ │ └── multi-targets
│ │ │ ├── freshen
│ │ │ ├── full
│ │ │ ├── quick
│ │ │ ├── stage1
│ │ │ └── strategy
│ │ │ └── seed
│ │ │ ├── stage1
│ │ │ └── stage3
│ │ └── ~funtoo
│ │ ├── build.conf
│ │ └── multi-targets
│ │ ├── freshen
│ │ ├── full
│ │ ├── quick
│ │ ├── stage1
│ │ └── strategy
│ │ └── seed
│ │ ├── stage1
│ │ └── stage3
│ ├── fslayouts
│ │ └── funtoo
│ │ ├── layout.conf
│ │ └── type
│ │ ├── image
│ │ └── snapshot
│ └── metro.conf
├── features
│ └── probe.conf
├── metro
├── misc
│ └── locale.gen
├── modules
│ ├── catalyst_support.py
│ ├── catalyst_support.pyo
│ ├── flexdata.py
│ ├── flexdata.pyo
│ ├── flexdata2.py
│ ├── flexdata3.py
│ ├── flexdata4.py
│ ├── targets.py
│ └── targets.pyo
├── scripts
│ ├── build.sh
│ ├── cleaner.sh
│ ├── dlstats.sh
│ └── ezbuild.sh
├── subarch
│ ├── amd64
│ │ ├── 2008.spec
│ │ └── 2009.spec
│ ├── amd64.spec
│ ├── athlon-xp.spec
│ ├── core2.spec
│ ├── core2_32.spec
│ ├── generic32.spec
│ ├── generic64.spec
│ ├── i686
│ │ ├── 2008.spec
│ │ └── 2009.spec
│ ├── i686.spec
│ ├── native32.spec
│ ├── native64.spec
│ ├── opteron-sse3.spec
│ ├── opteron.spec
│ ├── opteron_32.spec
│ ├── pentium4.spec
│ ├── phenom.spec
│ ├── x86.spec
│ ├── xen-nocona.spec
│ └── xen-prescott.spec
└── targets
└── gentoo
├── openvz.spec
├── snapshot
│ ├── common.spec
│ ├── global.spec
│ ├── source
│ │ ├── git
│ │ └── rsync
│ └── type
│ ├── dead
│ └── live
├── snapshot.spec
├── stage
│ ├── capture
│ │ ├── ami.spec
│ │ └── tar.spec
│ ├── common.spec
│ ├── files.spec
│ ├── stage3-derivative.spec
│ ├── stage3-generator.spec
│ └── steps.spec
├── stage1
│ └── strategy
│ ├── local
│ └── remote
├── stage1.spec
├── stage2
│ └── strategy
│ ├── local
│ │ ├── stage1
│ │ └── stage3
│ └── remote
│ ├── stage1
│ └── stage3
├── stage2.spec
├── stage3-freshen.spec
├── stage3-quick.spec
├── stage3.spec
└── stage4.spec
For the beginner, just retain that :
- Python is the master of the house, everything (or at least near everything) in Funtoo tools is written Python, and as being a Funtoo tool, Metro in itself plus all of the stuff it uses are written in Python.
- The first place to start with Metro tweaking are the .spec files located under targets/gentoo. Those files control what Metro does at each stage and are basically a melt of Metro directives and some Python embedded scripts and BASH embedded scripts. If something goes wrong.
- Another good place to look at is subarch : you will find here a couple of .spec files that governs how the build toolchain (libc, c/c++ compiler and related tools) will be generated, how they will be used (notice the CHOST variable) and at the end what will be put in the make.conf file in your stages archives.
The Metro Data Model page explains the logic used in spec files.
TBC
The GNU toolchain
"How the build toolchain will be generated", what does it mean ? As you may know, the GNU toolchain is a multiform set of tools that translate a source program written in a "high" level language like C, C++ or Fortan in a binary form understandable by the microprocessor of your computer. Moreover this set of tools do various sorts of things like source code grammar and syntax checking, code optimization, dead code removal, code instrumentation, and so on.
The multiform magic keyword means that the GNU toolchain can be compiled for many processors archicture, this of course include the well-known IA-32 (x86 family processors) but also IA-64 (Itaninum), PA-RISC (HPPA), Motorola 68k series, Dec Alpha, SPARC series, ARM, and many others including the so well known DEC PDP/11 series (technically speaking, with the adequate packages and keywordings, Funtoo could bring the steam power on a DEC PDP/11 machine and it could be possible to build stage 3 images for that platform).
Basically the GNU tool chains relies on three components :
- The so well-known GNU Compilers Collection (sys-devel/gcc is the Funtoo package that deploys this suite). A common mistake is to call GNU Compiler, but set that aside for now.
- Some related tools called behind the scene by the GNU Compiler Collection or required by the GNU autoconf/automake system upon which Funtoo relies on. Those behind the scene tools are for example the GNU linker (ld) or the the GNU Assembler (as). Under Funtoo, those tools are deployed by the package sys-devel/binutils.
- The GNU lib C : this one regroups a large set of system functions used by a large numbers of programs out there and contains functionalities like files manipulation, memory allocation, and so on. It can be barely seen as being a software interface (abstraction) between the system kernel and a program that lives in the userland world.
Note : the GNU libC is not the only one solution available in the Free Software world. Several alternatives like uLibC or newlib exist and can be used by the GNU Compiler Collection and the related tools it uses.
The GNU toolchain can be built as being :
- a native toolchain : the computer that runs the toolchain is the same that executes the binaries produced the toolchain, this is the most common case. (e.g. an i686 processor machine running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for the exact same architecture an i686 processor machine running GNU/Linux with the GNU Lib C).
- a cross-compilation chain : the computer that runs the toolchain is NOT the same that executes the produced the toolchain (e.g. an i686 processor running GNU/Linux with the GNU Lib C produces a toolchain that generates binaries for a SPARC machine running Solaris). Moreover, you can do funny brain-dead things like : generate a toolchain running on a PowerPC GNU/Linux system that produces another toolchain targeting a SPARC machine under Solaris which, in its turn, produces binaries targeting an Alpha machine running under FreeBSD).
Have you ever noticed ?
By CONVENTION, the GNU toolchain internal mechanisms REQUIRES to prefix all the toolchain executables by a quadruplet as below :
CPU-machine-OS-libC
- CPU : i486, i686, sparc, sparc64, alpha...
- machine : pc, sun4u, unknown...
- OS : linux, solaris, freebsd...
- libC : gnu (for glibc), ulibc...
This is stricly conventional but powerful as it allows the toolchain to easily detect when it has to be build it as a native toolchain or as a cross-compilation toolchain. Also note that some architectures like SPARC version 9 (i.e. UltraSPARC) uses a 64 bits kernel and 32 bits userland binaries (although it is possible to have 64 bits binaries, most of the time producing 64 bits binaries does not bring any advantage and worse, can degrade the performance on some machines due to additional memory read/write cycles required to get/write 64 bits values). This is exactly the reason why you have sys-devel/gcc and sys-devel/kgcc64, the later being only used to build the system kernel.
Now lets have a look on your machine at :
- the CHOST variable in your /etc/make.conf and what lies under your /usr/sbin.
- where points commands like /usr/bin/ld or /usr/bin/as
- what prefixes g++ or gcc in /usr/bin (take note that /usr/bin/g++ and /usr/bin/<CPU-machine-OS-libC>-g++ are exactly the same).
This is not a coincidence if the CHOST variable and the prefix used by the toolchain executables matches ;-)
How things works ?
Playing with the cross-compilation capablities of the GNU toolchain is out the scope of this article, just retain on this subject that it is mainly used with embedded systems which have too few resources to host a full compilation system or in build farms that generates binaries for several CPU architectures because it is now the time to dive in the main course of the action.
As you may have noticed in the Metro Quick Start Tutorial the beginning of the story lies in the script
TBC
Adding a new sub architecture
As of January 2011, Metro can handle the IA-32 architecture (x86/32 and x86/64 bits) as well as being able to handle several x86 flavous lying beneath the generic 'x86 and 'amd64' taxonomies. The example here is a try to add the necessary specification file to make it generate stage 1/2/3 tarballs for tiers architectures.