ReBootstrap

This page documents the effort to bootstrap Funtoo Linux using Aboriginal Linux.

What is Aboriginal Linux?

Aboriginal Linux's motto is "we cross compile so you don't have to". It replaces cross compiling with native compiling under an emulator (generally QEMU). Funtoo can use this to natively create stage 1/2/3 root filesystems on an arbitrary target, which greatly simplifies adding support for new architectures and regression testing existing architectures.

Aboriginal Linux creates a simple native development environment for each type of target hardware, building the smallest self-contained Linux system capable of rebuilding itself entirely from source code. This requires seven packages: linux, uClibc, busybox, binutils, gcc, make, and bash.

The resulting system can then boot under QEMU (or on appropriate hardware, if available) to provide a native development environment, eliminating the need for any further cross compiling. You can wget, configure, make, and install additional source packages inside the emulator.

Aboriginal Linux's web page is at http://landley.net/aboriginal and a long presentation about it is available at http://speakerdeck.com/u/mirell/p/developing-for-non-x86-targets-using-qemu

How do I use Aboriginal Linux?

Aboriginal Linux provides prebuilt native development environments for x86, x86-64, arm, mips, powerpc, and more, ready to run under the emulator. Install QEMU 1.0 or later, then go to http://landley.net/aboriginal/downloads/binaries", grab the appropriate system-image tarball, extract it, and run one of the following three shell scripts (included in the tarball):

• ./run-emulator.sh - Boot to a shell prompt, with the simplest configuration: a read-only (squashfs) root filesystem with a tmpfs mounted on /home.

• ./dev-environment.sh - Boot to a shell prompt with a full development environment. This wrapper around run-emulator.sh adds a 2 gigabyte ext2 /dev/hdb image mounted on /home (persistent writeable space, in large amounts), and allocates 256 megabytes of physical memory for the emulator (enough to run gcc building complex packages). If distccd and the appropriate target's cross compiler are available in the host's $PATH, this script also configures the emulated environment to call out to the cross compiler via distcc, transparently moving the heavy lifting of compilation outside of the emulator.

• ./native-build.sh - Run an automated build. This requires a "control image", which is a squashfs filesystem the emulated system sees as /dev/hdc mounted on /mnt. The init script in the emulated root filesystem checks for /mnt/init and if that exists, it runs that instead of providing a shell prompt. (Actually it asks the user to press a key if they want a shell prompt, with a three second timeout.) See "http://landley.net/aboriginal/control-images" for details.

(You can build your own system images from source by following the instructions at "http://landley.net/aboriginal", but ReBootstrap does not require this.)

What is an "LFS build"?

It's a Linux From Scratch root filesystem natively built on top of an Aboriginal Linux system image.

One of the example native-build.sh control-images Aboriginal Linux provides is "lfs-bootstrap.hdc", which natively compiles and installs (almost) all of the LFS chapter 6 packages under the emulated development environment, according to the LFS build instructions.

The control image copies the read-only root filesystem into a writeable directory under /home, chroots into that, builds packages and installs them over the existing (busybox) versions, and then tars up the resulting filesystem.

This provides a much more capable native build environment for the target. Since Aboriginal Linux can already provide this for the supported targets, Funtoo uses this as its starting point instead of starting from the minimal build environment and building/installing all of its own prerequisites.

The plan of attack is as follows:

1. Start from an LFS build, plus necessary stuff for Portage to run. (Git, python, portage tree, profile...)
   1. Use the i686 target first, so we can test quickly via chroot.
2. Create a package.provided file for the local LFS system so that Portage doesn't complain of unsatisfied /var/db/pkg entries.
3. Use Portage to build a stage1 tarball to /tmp/stage1root.

Once we have a stage 1 for the target, Funtoo has been bootstrapped and we can extend this approach to bootstrap Funtoo on any architecture.

Details

• Setup LFS-i686 filesystem

wget http://landley.net/aboriginal/downloads/binaries/extras/lfs-bootstrap.tar.gz-i686

tar xvzf lfs-bootstrap.tar.gz-i686

sudo env -i HOST=i686 $(which chroot) lfs-bootstrap /sbin/init.sh

• Install python

wget http://python.org/ftp/python/2.7.2/Python-2.7.2.tar.bz2

tar xf Python-2.7.2.tar.bz2

cd Python-2.7.2

./configure --prefix=/usr

make -j $CPUS

make install

• Install git

wget http://git-core.googlecode.com/files/git-1.7.8.2.tar.gz

tar xzvf git-1.7.8.2.tar.gz

cd git-1.7.8.2

./configure --without-tcltk --without-python --without-iconv

make NO_PERL=1

make NO_PERL=1 install

• Install portage tool

cd /usr/lib

git clone git://github.com/funtoo/portage-funtoo portage

cd portage

mkdir -p /usr/share/portage

ln -s /usr/lib/portage/cnf /usr/share/portage/config

for i in emerge ebuild; do ln -s /usr/lib/portage/bin/$i /usr/bin/$i; done

• Install portage tree, setup profiles

cd /usr

tar xvzf portage_tarball.tar.gz

mv portage-system portage

ln -s /usr/portage/profiles/default/linux/$ARCH/2008.0 /etc/make.profile

• setup package.provided

git clone git://github.com/funtoo/funtoo-overlay /var/tmp/funtoo-overlay

mkdir -p /etc/portage

cp /var/tmp/funtoo-overlay/funtoo/scripts/stage3.provided /etc/portage/package.provided

• Try to emerge something. Notice it doesn't work. Frown at it.
  • package.provided doesn't support USE flags. What the...?

Prerequisites

1. A smaller Portage tree to not eat up so much filesystem space. This has now been implemented. The script /root/git/funtoo-overlay/funtoo/scripts/generate-system-tree.py can be used to generate a system-only Portage tree.
   1. The dependencies are still circular and strange.
      1. Libtool is an abomination. It exists to make non-elf systems behave like ELF, and since Linux switched from a.out to ELF in 1996 (when it was 4 years old) there's nothing for Libtool to actually DO on Linux, yet it still manages not to do it properly. Literally the ONLY thing installing Libtool on a Linux system does is introduce build breaks. (Thank you, Free Software Foundation.)
2. A package.provided list of a current Funtoo system. This can be found at /root/git/funtoo-overlay/funtoo/scripts/stage3-provided.txt.
3. The profiles aren't target-agnostic. There's no way to go

"build for this host, I don't really care what it is".

1. 1. You need an existing toolchain to build anything, and if it's gcc its "tuple" is available via "gcc -dumpmachine", so this is queryable at runtime.
   2. See [1] for ongoing work to genericize this.
      1. the keywords=* stuff helps greatly, but make.conf moved into profiles and is still somewhat target-specific. More cleanup needed here. How do you build on an arbitrary host without human intervention (from a cron job)?

Status

The approach above didn't seem to correctly provide needed dependencies. Options at this point:

1. Bundle up what we have as a stage3 and see if Metro's steps are more successful at using it as a stage3 (we were trying simple emerge commands rather than the explicit metro steps for testing)
   1. When you cd into the profile directory and emerge an ebuild file directly, it generally builds. It's dependency calculation that's screwed up: package.provided injects a blank package with no USE flags, and anything that depends on a package with a USE flag tries to rebuild that package. I.E. package.provided is essentially useless.
      1. drobbins suggested something like "portage inject packagename USE="flag flag flag" [files...]" to retroactively notify portage of an installed package. (Aboriginal Linux has a BINARY_PACKAGE_TARBALLS option using "touch timestamp; build package; find output -newer timestamp", and this could be inserted in the LFS build fairly easily. Or just use a blank file list and not care that portage can't upgrade/uninstall the package, we just need the dependency generation to work.)
2. If package.provided is not working, simply copy /var/db/pkg from an existing stage3 for a massive injection of fake package information.
3. Or the option we are going to go with, which is stop trying to build a stage3, and build a minimal stage1 that we will feed to metro to take care of the rest.