ZFS as Root Filesystem

Introduction

This wiki will show you how to install Funtoo on ZFS (rootfs).

Prerequisites

   Important

ZFS designed for 64-bit systems. We only recommending and supporting 64-bit platforms and installations!

ZFS recommendation is to control entire disk, hence, guide reflecting only installing ZFS on whole disk and legacy boot. Installing on UEFI requires separate partition for /boot, formatted into FAT32 and not covered here, though, installation on UEFI is certainly possible. Guide also not describing anything related to encryption.

Downloading the ISO (With ZFS)

In order to install Funtoo on ZFS, you will need an environment such as live media with ZFS tools provided.

Creating a bootable USB from ISO (From a Linux Environment)

After you download the iso, you can do the following steps to create a bootable USB: TBA.

Booting the ISO

(example using ubuntu live iso 16.04.1 amd64)

ubuntu@ubuntu:~$ sudo -i
root@ubuntu:~# apt-add-repository universe
root@ubuntu:~# apt update
root@ubuntu:~# apt install --yes debootstrap gdisk zfs-initramfs

verify that the ZFS kernel module has loaded

root@ubuntu:~#  dmesg | grep ZFS
[  377.595348] ZFS: Loaded module v0.6.5.6-0ubuntu10, ZFS pool version 5000, ZFS filesystem version 5

Drive Partitions

We are letting ZFS automatically partition the drive. This ideal for our example using a single disk and legacy (BIOS) boot.

Creating of pool

Create a ZFS Storage Pool using a single whole disk

root@ubuntu:~# zpool create -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -O atime=off -m none -R /mnt/funtoo rpool /dev/disk/by-id/foo

The options used here are

option	description
create	Use zpool to create a ZFS Storage Pool.
-f	Force the use of the selected disk.
-o ashift=12	Alignment of the pool to underlying device sectors. The recommended value is 12, which corresponds to 2^12 Bytes or 4 KiB. This value is typical of drives nowdays. Can only be set at pool creation.
-o cachefile=/tmp/zpool.cache	Create a pool configuration cache and place it in /tmp. This will be required for our Funtoo install.
-O normalization=formD	Recommended. Set the default Unicode (UTF-8) normalization for future filesystems (created within this pool) to 'formD'.
-O atime=off	Optional. As a default preference, set future filesystems (created within this pool) to not update file access time. Useful if wanting to reduce writes to disk (e.g. Solid State Drives). Can cause problems for mailers and other software relying on file access-time data.
-m none	Do not set mountpoint for this storage pool. (This guide will address this later).
-R /mnt/funtoo	Alternate root directory. Essentially a temporary 'mount point' for our pool.
rpool	The name of this ZFS Storage Pool. The pool name is irrelevant, however rpool will be used throughout this guide.
/dev/disk/by-id/foo	The path to the physical disk. e.g. /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF. Known in 'ZFS-speak' as a 'VDEV'.

Please note that the options 'ashift' and 'cachefile' are preceded with a lowercase '-o', while 'normalization' and 'atime' are preceded with an uppercase '-O'.

Without additional options this will create our storage pool and enable all 'features' available under version 0.6.5.6. The pool will be automatically mounted at the (temporary) location /mnt/funtoo.

To confirm the presence of our newly created pool

root@ubuntu:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME                                             STATE     READ WRITE CKSUM
	rpool                                            ONLINE       0     0     0
	  ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF  ONLINE       0     0     0

errors: No known data errors

Create ZFS Datasets

Now we will create one or more ZFS filesystems within our storage pool. These will contain Funtoo Linux.

Create the Root file system (Required)

root #  zfs create -o mountpoint=none -o canmount=off rpool/ROOT
root #  zfs create -o mountpoint=/ rpool/ROOT/funtoo

Create optional Datasets

Described below datasets are examples and optional. It's up to users to perform creation of their own datasets.

Home

root #  zfs create -o mountpoint=/home rpool/HOME

Portage directories

root #  zfs create -o mountpoint=none -o canmount=off -o setuid=off rpool/FUNTOO
root #  zfs create -o mountpoint=/usr/portage rpool/FUNTOO/portage
root #  zfs create -o mountpoint=/usr/portage/distfiles rpool/FUNTOO/distfiles

Build directory

root #  zfs create -o mountpoint=/var/tmp/portage -o compression=lz4 -o sync=disabled rpool/FUNTOO/build

To confirm the presence of the filesystems that we have created

root@ubuntu:~# zfs list -t all
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool               660K  19.3G    96K  none
rpool/HOME           96K  19.3G    96K  /mnt/funtoo/home
rpool/ROOT          192K  19.3G    96K  none
rpool/ROOT/funtoo    96K  19.3G    96K  /mnt/funtoo

Make the root filesystem bootable

   Important

Do not skip this!

When booting from ZFS, you must specify a boot device and a root file system within the pool that was identified by the boot device. By default, the dataset selected for booting is the one identified by the pool's bootfs property.

root # zpool set bootfs=rpool/ROOT/funtoo rpool

Installing Funtoo

Now that initial ZFS pool created, datasets ready, we can perform stage3 unpack and basic system installation. This part does not differ much from regular Funtoo Linux installation. http://www.funtoo.org/Install#Installing_the_Stage_3_tarball. We recommend using funtoo-current stages for ZFS.

root #  cd /mnt/funtoo
root #  wget http://build.funtoo.org/funtoo-current/x86-64bit/generic_64/stage3-latest.tar.xz

Extract the contents with the following command, substituting in the actual name of your stage 3 tarball (in case of arch optimized stage).

root #  tar xpf stage3-latest.tar.xz

Now we need to create chroot environment by following:

root # cd /mnt/funtoo
root # mount -t proc none proc
root # mount --rbind /sys sys
root # mount --rbind /dev dev

Important step is to copy ZFS cache we created at the very beginning of our rpool creation into chroot.

root # mkdir -p /mnt/funtoo/etc/zfs
root # cp /tmp/zpool.cache /mnt/funtoo/etc/zfs/zpool.cache

You'll also want to copy over resolv.conf in order to have proper resolution of Internet hostnames from inside the chroot:

root # cp /etc/resolv.conf /mnt/funtoo/etc/

We are ready to chroot

root # chroot /mnt/funtoo /bin/bash
root # export PS1="(chroot) $PS1"; cd

Configure your system according to the main Install Guide. During startup ZFS Filesystems will be mounted without the need of entries in /etc/fstab. Other partitions such as CD-ROMs, tmpfs, etc. should still be included in /etc/fstab.

Next, update the portage tree:

root # emerge --sync
root # env-update
root # source /etc/profile

Installing ZFS userspace and bootloader

Installing the ZFS userspace tools and kernel modules

   Warning

At this time (2016-10-03) ZFS versions 0.6.5.4 and greater are hard masked. The kernel included with the latest funtoo-current stage3 is (debian-sources) linux 4.5.2. ZFS version 0.6.5.6 or higher is required for 4.5 kernels. The following instructions allow the use of ZFS with this stage3 kernel. This however requires the installation of software currently masked, pending validation testing.

Until validation has been completed, the use of hard masked software in a production environment is not recommended.

Using your favourite editor, create and edit the file /etc/portage/package.unmask so that it contains

   /etc/portage/package.unmask - example using ZFS 0.6.5.7

=sys-fs/zfs-0.6.5.7
=sys-fs/zfs-kmod-0.6.5.7
=sys-kernel/spl-0.6.5.7

Again using your favourite editor, create and edit a second file, /etc/portage/package.accept_keywords so that it contains

   /etc/portage/package.accept_keywords - example using ZFS 0.6.5.7

=sys-fs/zfs-0.6.5.7 **
=sys-fs/zfs-kmod-0.6.5.7 **
=sys-kernel/spl-0.6.5.7 **

Install the ZFS packages, syncing the portage tree first if required:

root # emerge --sync
root # emerge --ask sys-fs/zfs

Once successfully merged, add the following services to the boot runlevel of OpenRC:

root # rc-update add zfs-import boot
root # rc-update add zfs-mount boot

Add another two services, this time to the default runlevel:

root # rc-update add zfs-share default
root # rc-update add zfs-zed default

Create a ZFS-friendly initramfs

The Funtoo stage3 includes a linux kernel and initramfs. The initramfs is designed to mount and start Funtoo Linux on a variety of file systems. The initramfs contained within the stage3 will NOT mount and start Funtoo in our ZFS storage pool. We must create an updated 'ZFS-friendly' initramfs.

Optional: Update to the latest No results:

root # emerge --oneshot sys-kernel/genkernel

Use genkernel to create an initramfs capable of mounting our ZFS Storage Pool via the --zfs switch. Adjust --makeopts according to your available threads:

root # genkernel initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs
root ##g##* Funtoo Linux Genkernel; Version 3.4.40.11-funtoo##!g##
root ##g##*##!g## Running with options: initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs

root ##g##*##!g## Linux Kernel 4.5.2-1 for x86_64...
root ##g##*##!g## .. with config file /usr/src/linux-debian-sources-4.5.2/.config
root ##g##*##!g## busybox: >> Using cache
root ##g##*##!g## initramfs: >> Initializing...
root ##g##*##!g##         >> Appending base_layout cpio data...
root ##g##*##!g##         >> Appending auxilary cpio data...
root ##g##*##!g##         >> Copying keymaps
root ##g##*##!g##         >> Appending busybox cpio data...
root ##g##*##!g##         >> Appending modules cpio data...
root ##g##*##!g##         >> Appending zfs cpio data...
cp: cannot stat ‘/etc/zfs/zdev.conf’: No such file or directory
root ##y##*##!y## Could not copy file /etc/zfs/zdev.conf for ZFS
root ##g##*##!g##         >> Appending blkid cpio data...
root ##g##*##!g##         >> Appending modprobed cpio data...
root ##g##*##!g##         >> Compressing cpio data (.xz)...

root ##y##*##!y## WARNING... WARNING... WARNING...
root ##y##*##!y## Additional kernel cmdline arguments that *may* be required to boot properly...
root ##y##*##!y## add "dozfs" for ZFS volume management support
root ##y##*##!y## add either "real_root=ZFS" (bootfs autodetection) or "real_root=ZFS=<dataset>" to boot from a ZFS dataset

root ##g##*##!g## Do NOT report kernel bugs as genkernel bugs unless your bug
root ##g##*##!g## is about the default genkernel configuration...
root ##g##*##!g## 
root ##g##*##!g## Make sure you have the latest ~arch genkernel before reporting bugs.

Confirming the presence of the new initramfs:

root # ls /boot/*genkernel*
/boot/initramfs-genkernel-x86_64-4.5.2-1

Installing GRUB 2

GRUB 2 must be built with support for ZFS Storage Pools on a single disk. This is achieved using the 'libzfs' USE flag.

root # echo "sys-boot/grub libzfs" >> /etc/portage/package.use
root # emerge grub:2

Configuring the Bootloader

When zpool created our storage pool (rpool), it created partitions under a GPT scheme. In order to boot Funtoo Linux on a GPT partion under legacy (BIOS) boot, sys-boot/grub requires a small partition, called BIOS boot partition. By design, ZFS (zpool) left a very small unpartitioned space at the beginning of the disk. We will use sgdisk, which is part of sys-apps/gptfdisk to format this free space into a BIOS boot partition.

root # sgdisk --new=2:48:2047 --typecode=2:EF02 --change-name=2:"BIOS boot partition" /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF

A quick check to verify that GRUB 2 sees/supports ZFS:

root # touch /etc/mtab
root # grub-prob /
zfs

Installing GRUB2 to disk is as easy as:

root # grub-install /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF
Installing for i386-pc platform.
Installation finished. No error reported.

Now it's time for us to create grub's configuration file. First we must edit a few GRUB 2 settings in /etc/default/grub:

   Important

The following is required to boot Funtoo Linux on ZFS!

Replace the line

   /etc/default/grub - before

#GRUB_CMDLINE_LINUX=""

with

   /etc/default/grub - after

GRUB_CMDLINE_LINUX="dozfs real_root=ZFS=rpool/ROOT/funtoo"

Now create the configuration file GRUB 2:

root # grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/kernel-debian-sources-x86_64-4.5.2-1
Found initrd image: /boot/initramfs-genkernel-x86_64-4.5.2-1
done

A quick check to verify that the GRUB ZFS module is in place:

root # ls /boot/grub/*/zfs.mod
/boot/grub/i386-pc/zfs.mod

Final configuration

Configure your the Network according to of the Installation Guide.

Set the root password.

(chroot) # passwd

Exit the chroot and export your ZFS Storage Pool.

(chroot) # exit
root # umount -lR {dev,proc,sys}
root # cd /
root # zpool export rpool

Restart to boot into Funtoo Linux on ZFS Root!

After reboot

Snapshot

Take a snapshot of your Funtoo at installation.

root # zfs snapshot rpool/ROOT/funtoo@install

Further Reading

ZFS on Linux - http://www.zfsonlinux.org

ZFS - Gentoo Wiki - https://wiki.gentoo.org/wiki/ZFS

How to use Ubuntu 16.04 live CD with ZFS - https://github.com/zfsonlinux/zfs/wiki/Ubuntu-16.04-Root-on-ZFS

Troubleshooting

Forgot to reset password?

Restart the live CD/USB and repeat the process to install and load ZFS kernel modules.

This will most likely import the ZFS Storage Pool (rpool) and attempt to mount the various ZFS filesystems. Mounting the root partition (/) will fail as this location is not empty - / is allocated to the live CD/USB distribution!

root # zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME                                             STATE     READ WRITE CKSUM
	rpool                                            ONLINE       0     0     0
	  ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF  ONLINE       0     0     0

errors: No known data errors

Manually export the pool.

root # zpool export rpool
root # zpool list
no pools available

Import your storage pool using the following command. This will place the pool at /mnt/funtoo.

root # zpool import -o cachefile=/tmp/zpool.cache -R /mnt/funtoo -d /dev/disk/by-id/ rpool

Follow the previous set of instructions to chroot into your Funtoo Installation on ZFS root.