User:Invakid404/CLFS

CLFS with musl

This page covers the steps of getting a working CLFS "temporary environment" with musl instead of glibc and latest everything.

Setting up the environment (Chapter 2)

Instead of creating a partition, we'll build everything in a directory in our home. Let's assign it to a variable called `CLFS` and create it:

user $ export CLFS="$HOME/clfs"
user $ mkdir -v "${CLFS}"
mkdir: created directory '/home/invakid404/clfs'

We'll also need to create three subdirectories in our CLFS directory:

• cross-tools: this is where the cross compiler will go
• tools: this is where the so-called "temporary system" will go
• sources: this is where we'll download and build all packages

user $ mkdir -v "${CLFS}"/cross-tools
mkdir: created directory '/home/invakid404/clfs/cross-tools'
user $ mkdir -v "${CLFS}"/tools
mkdir: created directory '/home/invakid404/clfs/tools'
user $ mkdir -v "${CLFS}"/sources
mkdir: created directory '/home/invakid404/clfs/sources'

Let's also define some build-related environment variables:

• CLFS_HOST: the target triple of the host machine; modified to contain "cross" to handle the case where we're cross-compiling for the same target
• CLFS_TARGET: the target triple of the target architecture
• CLFS_ARCH: the name of the target architecture; used specifically when installing Linux kernel headers
• CLFS_CFLAGS: extra flags we'll pass to the GCC cross compiler
• MAKEFLAGS: flags we want to pass to GNU make by default

The host variable is straightforward:

user $ export CLFS_HOST=$(echo ${MACHTYPE} | sed -e 's/-[^-]*/-cross/')
user $ echo "${CLFS_HOST}"
x86_64-cross-linux-gnu

The target, arch, and CFLAGS depend on the target architecture you're going for. Here's a table of all currently tested architectures and their respective build variables:

Build variables per architecture

Architecture	CLFS_TARGET	CLFS_ARCH	CLFS_CFLAGS
x86_64	x86_64-unknown-linux-musl	x86_64	-m64
aarch64	aarch64-unknown-linux-musl	arm64

The rest of this page will assume aarch64, but the steps should be analogous for all listed architectures:

user $ export CLFS_TARGET="aarch64-unknown-linux-musl"
user $ export CLFS_ARCH="arm64"
user $ export CLFS_CFLAGS=""

For the makeflags, we'll tell GNU make to run as many jobs in parallel as we have threads to speed up the compilation:

user $ export MAKEFLAGS="-j$(nproc) -l$(nproc)"

Note that one can put these exports in a file and source them every time one restarts their session to avoid having to setting them manually each and every time. The same applies for all convenience environment variables we'll define later on.

Let's also set our PATH to something more restricted to reduce the influence of the host machine to a minimum. We'll also need to add our cross-tools directory to PATH to be able to use the cross compiler later:

user $ export PATH="${CLFS}/cross-tools/bin:/bin:/usr/bin"

Building the cross compiler (Chapter 3)

To get a cross compiler going, we'll need to do the following steps:

• Install the Linux headers for the target architecture
• Build binutils for the target architecture
• Build a static version of GCC without a standard library
• Build musl with the static version of GCC
• Build GCC again, now with musl as the standard library

Note that the official CLFS guide suggests building things like GMP, MPFR, and MPC separately, but we'll extract those in the GCC source directory and leave it up to the GCC makefile to do the heavy-lifting for us for simplicity. We'll also skip over ISL since it's not required and not too interesting for us at this stage.

Linux headers

First things first, let's fetch the Linux kernel sources. Throughout the rest of this page, we'll be putting the versions of packages in environment variables for convenience like so:

user $ export LINUX_VERSION="5.16.10"

Let's cd into the sources directory and download Linux:

user $ cd "${CLFS}"/sources
user $ wget https://cdn.kernel.org/pub/linux/kernel/v$(echo "${LINUX_VERSION}" | cut -d'.' -f1).x/linux-"${LINUX_VERSION}".tar.xz
user $ tar xvf linux-"${LINUX_VERSION}".tar.xz

Now, let's install the Linux headers for our target architecture. We'll install those in the tools directory to use in the temporary environment as well:

user $ pushd linux-"${LINUX_VERSION}"
user $ make mrproper
user $ make ARCH=${CLFS_ARCH} INSTALL_HDR_PATH=${CLFS}/tools headers_install
user $ popd

Binutils

Let's fetch and extract the binutils sources:

user $ export BINUTILS_VERSION="2.38"
user $ wget https://ftp.gnu.org/gnu/binutils/binutils-"${BINUTILS_VERSION}".tar.xz
user $ tar xvf binutils-"${BINUTILS_VERSION}".tar.xz

Now, let's configure binutils. The options we'll use are explained here. The only deviation we'll make is we'll prepend the prefix and lib path with the CLFS directory (this will be a reoccurring trend in this page since we aren't creating the symlinks in /) and we'll skip over some flags that aren't 100% necessary:

user $ mkdir binutils-build
user $ pushd binutils-build
user $ AR=ar AS=as ../binutils-${BINUTILS_VERSION}/configure \
  --prefix=${CLFS}/cross-tools \
  --host=${CLFS_HOST} \
  --target=${CLFS_TARGET} \
  --with-sysroot=${CLFS} \
  --with-lib-path=${CLFS}/tools/lib \
  --disable-nls \
  --disable-static \
  --enable-64-bit-bfd \
  --disable-multilib \
  --disable-werror

We're now ready to build and install it:

user $ make
user $ make install
user $ popd

Static GCC

Now, it's time to build a static GCC, which we'll use to build musl. For this, we'll need to fetch the sources of some GCC dependencies first, namely MPFR, GMP, and MPC:

user $ export MPFR_VERSION="4.1.0"
user $ export GMP_VERSION="6.2.1"
user $ export MPC_VERSION="1.2.1"
user $ wget https://ftp.gnu.org/gnu/mpfr/mpfr-"${MPFR_VERSION}".tar.xz
user $ wget https://ftp.gnu.org/gnu/gmp/gmp-"${GMP_VERSION}".tar.xz
user $ wget https://ftp.gnu.org/gnu/mpc/mpc-"${MPC_VERSION}".tar.gz

Note that we won't build them separately. Instead, we'll extract them inside the GCC source directory and let the GCC makefile handle that for us. With that out of the way, we should also fetch and extract the GCC sources:

user $ export GCC_VERSION="11.2.0"
user $ wget https://ftp.gnu.org/gnu/gcc/gcc-"${GCC_VERSION}"/gcc-"${GCC_VERSION}".tar.xz
user $ tar xvf gcc-"${GCC_VERSION}".tar.xz

Now, as mentioned above, we'll move into the GCC source directory and extract its dependencies inside:

user $ pushd gcc-"${GCC_VERSION}"
user $ tar xvf ../mpfr-"${MPFR_VERSION}".tar.xz && mv -v mpfr-"${MPFR_VERSION}" mpfr
user $ tar xvf ../gmp-"${GMP_VERSION}".tar.xz && mv -v gmp-"${GMP_VERSION}" gmp
user $ tar xvf ../mpc-"${MPC_VERSION}".tar.gz && mv mpc-"${MPC_VERSION}" mpc
user $ popd

We can get to building now. Let's create a separate directory for the GCC build and move into it:

user $ mkdir gcc-build
user $ pushd gcc-build

To configure GCC, we'll do a mixture of what's described in the regular and embedded CLFS books:

user $ AR=ar LDFLAGS="-Wl,-rpath,${CLFS}/cross-tools/lib" \
  ../gcc-"${GCC_VERSION}"/configure \
  --prefix=${CLFS}/cross-tools \
  --build=${CLFS_HOST} \
  --host=${CLFS_HOST} \
  --target=${CLFS_TARGET} \
  --with-sysroot=${CLFS} \
  --with-local-prefix=${CLFS}/tools \
  --with-native-system-header-dir=/tools/include \
  --disable-shared \
  --without-headers \
  --with-newlib \
  --disable-decimal-float \
  --disable-libgomp \
  --disable-libssp \
  --disable-libatomic \
  --disable-libitm \
  --disable-libsanitizer \
  --disable-libquadmath \
  --disable-libvtv \
  --disable-libcilkrts \
  --disable-libstdc++-v3 \
  --disable-threads \
  --disable-multilib \
  --enable-languages=c \
  --with-mpfr-include=$(pwd)/../gcc-"${GCC_VERSION}"/mpfr/src \
  --with-mpfr-lib=$(pwd)/mpfr/src/.libs

As mentioned in the book, we don't need to build the entirety of GCC at this stage. The following is enough:

user $ make all-gcc all-target-libgcc
user $ make install-gcc install-target-libgcc
user $ popd

With that, we should now have a somewhat functioning GCC cross compiler!

user $ "${CLFS}"/cross-tools/bin/"${CLFS_TARGET}"-gcc --version
aarch64-unknown-linux-musl-gcc (GCC) 11.2.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

musl

Now that we have a cross compiler, we can use it to build musl. As always, we'll start by fetching and extracting the sources:

user $ export MUSL_VERSION="1.2.2"
user $ wget https://musl.libc.org/releases/musl-"${MUSL_VERSION}".tar.gz
user $ tar xvf musl-"${MUSL_VERSION}".tar.gz
user $ pushd musl-"${MUSL_VERSION}"

Configuring musl is pretty straightforward. We'll do it pretty much exactly like in the embedded book.

user $ CC="${CLFS_TARGET}-gcc ${CLFS_CFLAGS}" \
./configure \
  CROSS_COMPILE=${CLFS_TARGET}- \
  --prefix=/ \
  --target=${CLFS_TARGET}

Note that we're also setting the "CC" environment variable with our cross compiler and the CFLAGS we've defined at the start. Compiling and installing musl is just as straightforward:

user $ make
user $ DESTDIR=${CLFS}/tools make install
user $ popd

We have musl installed now! There's a couple of small issues though...

The first one is subtle and although it won't be apparent for a while, it will bite us later, so let's look into it right now. If we peek into the lib directory, we'll notice that something is off:

user $ pushd "${CLFS}"/tools/lib
user $ ls -la
total 3616
drwxr-xr-x 2 invakid404 invakid404     270 Feb 23 09:42 .
drwxr-xr-x 4 invakid404 invakid404      32 Feb 23 09:42 ..
-rw-r--r-- 1 invakid404 invakid404    1736 Feb 23 09:42 Scrt1.o
-rw-r--r-- 1 invakid404 invakid404    2000 Feb 23 09:42 crt1.o
-rw-r--r-- 1 invakid404 invakid404    1072 Feb 23 09:42 crti.o
-rw-r--r-- 1 invakid404 invakid404    1016 Feb 23 09:42 crtn.o
lrwxrwxrwx 1 invakid404 invakid404      12 Feb 23 09:42 ld-musl-aarch64.so.1 -> /lib/libc.so
-rw-r--r-- 1 invakid404 invakid404 2709306 Feb 23 09:42 libc.a
-rwxr-xr-x 1 invakid404 invakid404  937488 Feb 23 09:42 libc.so
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 libcrypt.a
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 libdl.a
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 libm.a
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 libpthread.a
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 libresolv.a
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 librt.a
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 libutil.a
-rw-r--r-- 1 invakid404 invakid404       8 Feb 23 09:42 libxnet.a
-rw-r--r-- 1 invakid404 invakid404    2648 Feb 23 09:42 rcrt1.o

The dynamic linker symlink is incorrectly pointing at "/lib/libc.so", while it should be pointing at the libc.so in the current directory. Thankfully, this is an easy fix:

user $ unlink ld-musl-aarch64.so.1
user $ ln -sf libc.so ld-musl-aarch64.so.1
user $ ls -la ld-musl-aarch64.so.1
lrwxrwxrwx 1 invakid404 invakid404 7 Feb 23 09:46 ld-musl-aarch64.so.1 -> libc.so

And now it's fine. Note that we made the symlink relative instead of absolute, that way it'll work when we chroot as well.

The other issue we'll run into in the next step is, as you might have noticed from the command output above, the startfiles (crt*.o) are installed in tools instead of cross-tools. This would result in a linker error while building libgcc, so we'll need to create some symlinks to those in cross-tools to get things to work properly:

user $ find -type f \
  -iname 'crt*.o' \
  -exec ln -sf ../../../tools/lib/{} "${CLFS}"/cross-tools/"${CLFS_TARGET}"/lib/{} \;
user $ ls -la "${CLFS}"/cross-tools/"${CLFS_TARGET}"/lib
total 16
drwxr-xr-x 3 invakid404 invakid404    65 Feb 23 10:00 .
drwxr-xr-x 4 invakid404 invakid404    28 Feb 23 09:16 ..
lrwxrwxrwx 1 invakid404 invakid404    27 Feb 23 10:00 crt1.o -> ../../../tools/lib/./crt1.o
lrwxrwxrwx 1 invakid404 invakid404    27 Feb 23 10:00 crti.o -> ../../../tools/lib/./crti.o
lrwxrwxrwx 1 invakid404 invakid404    27 Feb 23 10:00 crtn.o -> ../../../tools/lib/./crtn.o
drwxr-xr-x 2 invakid404 invakid404 12288 Feb 23 09:16 ldscripts
user $ popd

This should be sufficient to get GCC to build properly in the next step.

Final GCC

Having done that, we're now ready to build GCC against our freshly built musl. We already have everything fetched and set up correctly, so let's just wipe the "gcc-build" directory clean and start over:

user $ pushd gcc-build
user $ rm -r *

We can now re-enable the long list of libraries we disabled when we built the static GCC, with a single exception: we'll need to keep libsanitizer disabled, since it doesn't seem to build correctly with musl due to the fact musl has no "fstab.h" header. Again, we'll use a mixture of what the embedded and regular CLFS books list as flags:

user $ AR=ar LDFLAGS="-Wl,-rpath,${CLFS}/cross-tools/lib" \
  ../gcc-"${GCC_VERSION}"/configure \
  --prefix=${CLFS}/cross-tools \
  --build=${CLFS_HOST} \
  --target=${CLFS_TARGET} \
  --host=${CLFS_HOST} \
  --with-sysroot=${CLFS} \
  --with-local-prefix=${CLFS}/tools \
  --with-native-system-header-dir=/tools/include \
  --with-lib-path=${CLFS}/tools/lib \
  --disable-nls \
  --disable-static \
  --enable-languages=c,c++ \
  --disable-multilib \
  --disable-libsanitizer \
  --with-mpfr-include=$(pwd)/../gcc-"${GCC_VERSION}"/mpfr/src \
  --with-mpfr-lib=$(pwd)/mpfr/src/.libs

Unlike the static GCC build, we have to build the entirety of GCC now:

user $ make AS_FOR_TARGET="${CLFS_TARGET}-as" \
  LD_FOR_TARGET="${CLFS_TARGET}-ld"
user $ make install
user $ popd

Building the basic tools (Chapter 4)

Now, we're ready to start getting some tools cross-compiled for our target architecture. For the sake of completeness, we'll install everything listed under "Constructing a Temporary System" in the CLFS book, even if not everything we build is strictly necessary later.

Using the cross compiler

To use the cross compiler, we'll need to set some environment variables. I personally find it nice to spawn a new instance of "bash" to scope those variables to be able to get rid of them later:

user $ env \
  TERM="${TERM}" \
  CC="${CLFS_TARGET}-gcc ${CLFS_CFLAGS}" \
  CXX="${CLFS_TARGET}-g++ ${CLFS_CFLAGS}" \
  AR="${CLFS_TARGET}-ar" \
  AS="${CLFS_TARGET}-as" \
  RANLIB="${CLFS_TARGET}-ranlib" \
  LD="${CLFS_TARGET}-ld" \
  STRIP="${CLFS_TARGET}-strip" \
  DYNAMIC_LINKER=$(eval echo "${CLFS}/tools/lib/ld-musl-*.so.1") \
  bash --rcfile <(cat ~/.bashrc 2>/dev/null; echo 'PS1="(cross) ${PS1}"')
user $ echo "${CC}"
aarch64-unknown-linux-musl-gcc

GMP

Unlike when building the cross compiler, we'll build and install GMP separately instead. We already have the GMP sources, so we'll just do a clean extract and get to building. The only thing we'll be consistently changing in the configure commands is the prefix, because we don't have a /tools symlink:

user $ tar xvf gmp-"${GMP_VERSION}".tar.xz
user $ pushd gmp-"${GMP_VERSION}"
user $ CC_FOR_BUILD=gcc \
./configure \
  --prefix=${CLFS}/tools \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET} \
  --enable-cxx
user $ make
user $ make install
user $ popd

MPFR

Pretty much the same as above:

user $ tar xvf mpfr-"${MPFR_VERSION}".tar.xz
user $ pushd mpfr-"${MPFR_VERSION}"
user $ ./configure \
  --prefix=${CLFS}/tools \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET}
user $ make
user $ make install
user $ popd

MPC

Same as above:

user $ tar xvf mpc-"${MPC_VERSION}".tar.gz
user $ pushd mpc-"${MPC_VERSION}"
user $ ./configure \
  --prefix=${CLFS}/tools \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET}
user $ make
user $ make install
user $ popd

ISL

While this is an optional dependency and we skipped it earlier, we'll build it now, because why not :D

Nothing special, just fetch and unpack the sources, configure, build and install:

user $ export ISL_VERSION="0.24"
user $ wget https://gcc.gnu.org/pub/gcc/infrastructure/isl-"${ISL_VERSION}".tar.bz2
user $ tar xvf isl-"${ISL_VERSION}".tar.bz2
user $ pushd isl-"${ISL_VERSION}"
user $ ./configure \
  --prefix=${CLFS}/tools \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET}
user $ make
user $ make install
user $ popd

Zlib

Nothing unusual yet again:

user $ export ZLIB_VERSION="1.2.11"
user $ wget https://zlib.net/zlib-"${ZLIB_VERSION}".tar.gz
user $ tar xvf zlib-"${ZLIB_VERSION}".tar.gz
user $ pushd zlib-"${ZLIB_VERSION}"
user $ ./configure \
  --prefix=${CLFS}/tools
user $ make
user $ make install
user $ popd

Binutils

No special steps here either:

user $ pushd binutils-build
user $ rm -r *
user $ ../binutils-${BINUTILS_VERSION}/configure \
  --prefix=${CLFS}/tools \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET} \
  --target=${CLFS_TARGET} \
  --with-lib-path=${CLFS}/tools/lib \
  --disable-nls \
  --enable-shared \
  --enable-64-bit-bfd \
  --disable-multilib \
  --enable-gold=yes \
  --enable-plugins \
  --with-system-zlib \
  --enable-threads
user $ make
user $ make install
user $ popd

GCC

This is where we'll need to do some patching to get things to work. We'll need to disable stdinc++ (refer to this bug):

user $ pushd gcc-"${GCC_VERSION}"
user $ sed -i \
  's|^RAW_CXX_FOR_TARGET="$CXX_FOR_TARGET|& -nostdinc++|' \
  configure
user $ popd

We'll also need to disable libstdcxx-pch and libsanitizer for similar reasons as before:

user $ pushd gcc-build
user $ rm -r *
user $ ../gcc-"${GCC_VERSION}"/configure \
  --prefix=${CLFS}/tools \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET} \
  --target=${CLFS_TARGET} \
  --with-local-prefix=${CLFS}/tools \
  --disable-multilib \
  --enable-languages=c,c++ \
  --with-system-zlib \
  --with-native-system-header-dir=/tools/include \
  --disable-libssp \
  --enable-install-libiberty \
  --disable-libstdcxx-pch \
  --disable-libsanitizer
user $ make AS_FOR_TARGET="${AS}" \
  LD_FOR_TARGET="${LD}"
user $ make install
user $ popd

Ncurses

We'll deviate from the CLFS book here as well. Let's fetch the sources first:

user $ export NCURSES_VERSION="6.3"
user $ wget https://ftp.gnu.org/gnu/ncurses/ncurses-"${NCURSES_VERSION}".tar.gz
user $ tar xvf ncurses-"${NCURSES_VERSION}".tar.gz
user $ pushd ncurses-"${NCURSES_VERSION}"

First things first, we need to instruct ncurses to use the cross-compiled strip instead of the host one:

user $ sed -i -e "s/INSTALL_OPT_S=\\\"-s/& --strip-program=${STRIP}/" configure

Then, we'll need to build both ncurses and ncursesw since things will fail down the line if we don't. Let's build the regular one first. Note that we're also enabling terminfo-related flags to ensure things like clearing the terminal are going to work when we chroot:

user $ mkdir ncurses-build
user $ pushd ncurses-build
user $ ../configure \
  --prefix=${CLFS}/tools \
  --with-shared \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET} \
  --without-debug \
  --without-ada \
  --enable-overwrite \
  --with-build-cc=gcc \
  --enable-overwrite \
  --with-default-terminfo-dir=${CLFS}/tools/share/terminfo \
  --with-termlib
user $ make
user $ make install
user $ popd

Similarly, we'll build ncursesw as well. Note that we're changing the includedir as well, as that's where other packages are going to expect the headers to be:

user $ mkdir ncursesw-build
user $ pushd ncursesw-build
user $ ../configure \
  --prefix=${CLFS}/tools \
  --with-shared \
  --build=${CLFS_HOST} \
  --host=${CLFS_TARGET} \
  --without-debug \
  --without-ada \
  --enable-overwrite \
  --with-build-cc=gcc \
  --enable-widec \
  --includedir=${CLFS}/tools/include/ncursesw \
  --with-default-terminfo-dir=${CLFS}/tools/share/terminfo \
  --with-termlib
user $ make
user $ make install
user $ popd
user $ popd

Bash

Nothing too interesting here either.

user $ export BASH_VERSION="5.1.16"
user $ tar xvf bash-"${BASH_VERSION}".tar.gz
user $ pushd bash-"${BASH_VERSION}"
user $ cat > config.cache << "EOF"
ac_cv_func_mmap_fixed_mapped=yes
ac_cv_func_strcoll_works=yes
ac_cv_func_working_mktime=yes
bash_cv_func_sigsetjmp=present
bash_cv_getcwd_malloc=yes
bash_cv_job_control_missing=present
bash_cv_printf_a_format=yes
bash_cv_sys_named_pipes=present
bash_cv_ulimit_maxfds=yes
bash_cv_under_sys_siglist=yes
bash_cv_unusable_rtsigs=no
gt_cv_int_divbyzero_sigfpe=yes
EOF
user $ ./configure \
    --prefix=${CLFS}/tools \
    --build=${CLFS_HOST} \
    --host=${CLFS_TARGET} \
    --without-bash-malloc \
    --cache-file=config.cache
user $ make
user $ make install
user $ popd