Netboot image creation for SPARC machines

If your SPARC machine has a dead CD/DVD ROM drive or no CD/DVD ROM drive at all or if, for various reasons, you wish to build a netboot image from scratch the following paragraphs should help you.

Concepts

Basically, a netboot image is the sum of :

• Some bootstrap code (loader)
• An uncompressed kernel image
• An optional ramdisk / initrd image.

When the machine is powered-on, a Power On Self Test (POST) of the machine is done then, if nothing has been found defective Open Boot checks what the variable boot-device look like and take appropriate actions. If the machine has been set to boot via the network several situations can happen :

• if Open Boot has no WAN boot (Open Boot prior 4.17.1 ?) support and no DHCP support, a RARP request is emitted on the network to get an IP address from a RARP server then the machine starts a TFTP session on the machine who provided the IP and request to download a file named after the provided IP address (e.g. 192.168.4.101 => C0A80465).

• if Open Boot has been set to boot from the network using a DHCP request (boot-device is set to net:dhcp or boot net:dhcp has been entered at the Open Boot prompt), the machine will grab its network parameters using a DHCP request then starts to download the file according the DHCP response (filename and TFTP server), or by default adopt the same behavior than above.

• if Open Boot has been setup to perform a WAN boot, it uses what is contained in the variable network-boot-arguments to determine what to download from which location and with what networking parameters.

Once the netboot image has been downloaded, Open Boot transfers the execution the netboot image loader, which, at its turn will transfer to the Linux kernel which will run /sbin/init (or whatever else specified through the kernel command line). On most of the Funtoo machines in the world, /sbin/init will launch Open RC at the end do various things.

Short story:

OpenBoot -> Netboot loader -> Linux kernel -> /sbin/init -> Open RC -> Login on a tty (agetty+login) or whatever else suitable

The process is very similar to what happens when booting the machine from its hard drive, nothing magic here. The form varies, but the philosophy remains the same.

Limitations

According to various sources on the internet and experience returns, it looks like that :

• sun4u machines (e.g. UltraSPARC I/II/III/IV servers and workstations) do not initialize the processor MMU and thus, this requires the whole netboot image to fit within 4 megabytes. What happens if this rule is enforced? Simple: as the buffer where the netboot image is loaded into acts as a 4 megabytes ring-buffer, any byte exceeding the limit will overwrite the previously netboot image loaded data leading to the so famous "Fast Data Access MMU Miss" a couple of seconds after the machine started to download the netboot image.

• sun4v machines (e.g. UltraSPARC T1 and later) initialize their MMU and thus, are not subject to the above limitation. Pay attention to the startup messages when you power on the machine you will see something like "Init MMU" (assuming you are connected to the virtual console from ALOM):

...
0:0>VBSC selecting POST MIN Testing.
0:0>POST enabling threads: ffffffff
0:0>VBSC setting verbosity level 2
0:0>Start Selftest.....
0:0>Master CPU Tests Basic....Done
0:0>Init MMU.....
...

Fortunately for us : SPARC machines relies on a very small set of hardware from an model to another (modulo extension cards).
Unfortunately for us : 2.6 kernels, tends to be big even with an aggressive strip down of features and aggressive size optimizations.

Producing the netboot image

Kernel configuration

It is highly suggested to use modules as much as possible. Of course don't forget to:

• pull in the correct modules and firmware to handle your network adapter :-)
• Activate the following in General setup:
  • Initial RAM filesystem and RAM disk (initramfs/initrd) support
  • Embedded system (Support initial ramdisks compressed... choices won't show up until that option is activated)
  • Support initial ramdisks compressed using XZ (or what suits the compression you use)
• Put the filesystem used for your RAM disk built-in (suggested: ext2, do not use cramfs unless you want a readonly filesystem).

Preparing a ramdisk image

You have two strategies for the initramfs image. You can either create it from:

1. A cpio archive of a live filesystem (so far we didn't succeed in producing a usable compressed cpio archive, the kernel panics on boot)
2. A minimalistic raw image created through dd and a loopback device

Raw image creation

First you need to create a flat file which can hold a few megabytes (we us 16 MB here but you can take less space), associate it to a loopback device and then create a filesystem on it (ext2 in our case):

root # dd if=/dev/zero of=/usr/src/root.img bs=1M count=16
root # losetup /usr/src/root.img /dev/loop0
root # mkfs.ext2 /dev/loop0
root # mkdir /mnt/netboot-rootfs
root # mount /dev/loop0 /mnt/netboot-rootfs

Structure creation

First, you need to create a bare minimal file system structure in a working directory of you choice, although /mnt/netboot-rootfs is used here, anything left at of your choice can be used. You will need :

• /bin
• /dev
• /dev/pts
• /etc
• /sbin
• /lib
• /usr/bin
• /usr/sbin
• /tmp

root # cd /mnt/netboot-rootfs
root # mkdir -p bin dev dev/pts etc sbin usr/bin usr/sbin tmp 

Second, copy your /etc/passwd (shadow password are used, so only the logins are stored here) and /etc/groups:

root # cd mnt/netboot-rootfs
root # cp /etc/passwd /etc/group etc

Although it not mandatory required to copy those two files, it will, however help BusyBox' mdev.

Populating /dev

You have two ways to populate /dev:

1. Let the kernel manages everything for you (devtmpfs): A temporary tmpfs containing special devices entries are created very early at system start up and are then moved in the /dev directory once the root filesystem has been mounted.
2. Manually create some entries in what will be included with the ramdisk and let mdev so the rest when the netboot image has been booted.

For the first alternative, just enable Maintain a devtmpfs filesystem to mount at /dev and Automount devtmpfs at /dev, after the kernel mounted the rootfs (see Device drivers / Generic Devices Options) in your kernel configuration. For the second alternative, just use mknod to create some basic devices which are:

Mandatory shared object files coming from /lib

In this example dynamically linked binaries are used so the shared objects files their refer to must be copied. As of writing you must copy the following files from /lib to the lib directory located in your working space

• libblkid.so.1.1.0
• libblkid.so.1 (symlink to libblkid.so.1.1.0)
• libc-2.11.2.so
• libc.so.6 (symlink to libc-2.11.2.so)
• libcom_err.so.2.1
• libcom_err.so.2 (symlink to libcom_err.so.2.1)
• libcrypt.so.1
• libe2p.so.2
• libext2fs.so.2
• ld-2.11.2.so
• ld-linux.so.2 (symlink to ld-2.11.2.so)
• libm.so.6
• libpthread.so.0
• libutil.so.1
• libuuid.so.1
• libz.so.1.2.5
• libz.so.1 (symlink to libz.so.1.2.5)

root # cd /mnt/netboot-rootfs
root # cd /lib
root # cp libc-2.11.2.so libc-2.11.2.so ld-2.11.2.so ld-linux.so.2 libblkid.so.1.1.0 ..... /mnt/netboot-rootfs/lib

You can safely ignore libpam and libnss (if you don't copy the later, you will have to refer the remote hosts by their IP addresses only as libnss handles names resolution facilities). libpam will be unused because BusyBox won't be compiled with Pluggable Authentication Modules support.

Configuring and emerging BusyBox

BusyBox No results is a Swiss Knife that regroup in a single compact binary more than 30 well-known utilities (grep, ls, fdisk....). BusyBox does not mimic 100% of the original tools functionalities but supports the most used ones, as such it fits will memory constrained environments such as embedded systems or everywhere else where every byte of code loaded in memory matters. As such it fits very well to a netboot image.

A recommended configuration for BusyBox can be found in the talk section of this page. Just copy the recommended settings in /etc/portage/savedconfig/sys-apps/busybox-1.18.3 (assuming sys-apps/busybox-1.18.3 is the version available on your system).A few various points on suggested BusyBox configuration:

• No "Standalone shell" (CONFIG_FEATURE_SH_STANDALONE): if this option would be set, BusyBox will give the priority to the internal applets having the same name than the external commands (standalone scripts or binaries) you would add to your minimalist environment. A way to circumvent would be to launch the external commands with their absolute path but it is not a convenient way to use from a final user point of view.

• No support for ext2 utilities: As of writing, the BusyBox ext2 filesystem creation applet has an unwanted behaviour that makes it overwrite the first 1024 bytes of a partition. Creating an ext2 filesystem on the first partition of a hard drive starting at cylinder 0 (required if you want to boot your machine from it) with this applet would simply overwrite the Sun disklabel on the drive, thus killing the partition table on it.

• No support for ReiserFS: As of writing, the BusyBox ReiserFS is experimental and showed us some glitches.

• No support for fdisk: As of writing, the BusyBox fdisk applet is unable to generate proper Sun disklabels, thus making OpenBoot protest if you try too make to boot from the hard drive.

• No support for daemons like http, telnetd, dhcpd, ftpd or ntpd (not exhaustive) : this is a client only environment.

The time has now come to emerge BusyBox, this is achieved by:

root # CFLAGS="-mcpu=v9 -O2 -pipe" FEATURES="nodoc noinfo noman" USE="-pam make-symlinks mdev savedconfig" \ 
emerge -1 --root=/mnt/netboot-rootfs sys-apps/busybox

Here we are asking Portage to:

• Use generic SPARC v9 optimizations (all SPARC v9 processors share the same instruction set modulo specific extensions that are not in the SPARC v9 specification like VIS or VIS II)
• Not deploy any manpages, infopages or any kind of docuementation (see /usr/share) etc.
• Not use PAM (netboot environment is single administrative user only)
• Make symlinks for enabled BusyBox applets in the bin and sbin subdirectories of /mnt/netboot-rootfs (BusyBox is a multicall binary)
• Use /mnt/netboot-rootfs as the root directory for the deployment
• Not register sys-apps/busybox in the world file of the Funtoo "mini-instance" located under /mnt/netboot-rootfs /mnt/netboot-rootfs/var/lib/portage/world)

The backslash and the 2 lines form of the command line is present only for the sake of clarity, you can put everything on a single (long) line.

Putting the kernel modules and firmware images inside the ramdisk image

   Note

You will need to pass though this section even your kernel has no support for modules because some device drivers in the kernel need a firmware image (this is the case with Broadcom Tigeon 3 network adapters or QLogic Fibre Channel HBA for example).

Go in your kernel source directory and issue the following command:

root # cd /usr/src/linux
root # make modules_install INSTALL_MOD_PATH=/mnt/netboot-rootfs

Adding extra mandatory libs

If you add extra binaries to your netboot RAM disk, your are likely due to copy extra shared objects coming for /lib (and / or /usr/lib). To know the dependencies of your binary program, just run ldd on it:

root # ldd /sbin/fdisk
        libblkid.so.1 => /lib/libblkid.so.1 (0xf7f68000)
        libc.so.6 => /lib/libc.so.6 (0xf7de8000)
        libuuid.so.1 => /lib/libuuid.so.1 (0xf7dd0000)
        /lib/ld-linux.so.2 (0xf7fa4000)

If you see the output "VDSO" do not search for the file existence, VDSO stands for Virtual Dynamic Shared Object (this virtual shared object is provided by the Linux kernel itself through the remapping of some its code in the userland).

Compressing the initramfs image

Now unmount the raw image and compress it with xz:

root # cd /usr/src
root # umount /mnt/netboot-rootfs
root # xz --check=none --lzma2=dict=1MiB root.img
root # mv root.img.xz root.img

Because we use the xz compression must use :

• A CRC32 or no CRC at all (like here)
• A reasonable dictionary size (kernel documentation talks about a dictionary having no more than 1 Mbyte in size, see Documentation/xz.txt).

   Note

You must use root.img as the filename and you must put it under /usr/src, else the TFTP image construction process will fail.

Gluing things together

The root.img is seated, the kernel is rebuilt, now the most exciting part begins: generate the final netboot image. You have several options to get the job being done:

1. Use the command tilo which comes with sys-boot/silo
2. Use the facilities provided with the Linux kernel source code

At date of writing (February 2011) tilo has a bug that makes one the internal commands it uses to segfault when building over-sized images (> 4 Mbytes), so let's use the Linux kernel black magic. Simply :

• Go into your /usr/src/linux (Again: simply go there, do not change anything or rebuild a brand new kernel) and run the following:

root # make tftpboot.img

Various things will be checked and after a couple of seconds a file named tftpboot.img is produced in arch/sparc/boot.

What to do next?

Simply copy the freshly produced tftpboot.img into the appropriate directory of your TFTP server and, if you use symlinks to refer the image, make sure they points to the right file.

That's it!

At system start-up pay attention to :

....
[   81.921360] Trying to unpack rootfs image as initramfs...
[   81.922859] rootfs image is not initramfs (no cpio magic); looks like an initrd
[   81.943133] Freeing initrd memory: 3846k freed
...
[  103.079351] RAMDISK: xz image found at block 0
[  107.384859] VFS: Mounted root (ext2 filesystem) on device 1:0.
....

Strategy suggestion

As netboot images are required to be as compact as possible a 3 phases strategy is suggested:

1. Build a minimalist netboot (you barely need your network card driver, an IP stack, plus the required filesystems for step 2)
2. This minimalist netboot will grab another ramdisk image (using NFS for example), eventually unpack it and put it in the VFS (using Union FS for example) which include everything you need like kernel modules you IDE/SCSI/SAS controllers or additional filesystems
3. Once everything is in place, just exec what you need (i.e. Open RC, a set of homebrew scripts...)