注意：

The Funtoo Linux project has transitioned to "Hobby Mode" and this wiki is now read-only.

Difference between revisions of "ZFS as Root Filesystem"

From Funtoo
Jump to navigation Jump to search
 
(241 intermediate revisions by 13 users not shown)
Line 1: Line 1:
{{warning|This page is unofficial. ZFS as root filesystem is not supported under Funtoo Linux, mainly because it has limited benefit. ZFS is still supported but boot on a non-ZFS filesystem first. See [[ZFS]].}}
== Introduction ==
== Introduction ==


This tutorial will show you how to install Funtoo on ZFS (rootfs). This tutorial is meant to be an "overlay" over the [[Funtoo_Linux_Installation|Regular Funtoo Installation]]. Follow the normal installation and only use this guide for steps 2, 3, and 8.
This wiki article will show you how to install Funtoo on ZFS (rootfs).
 
=== Introduction to ZFS ===
 
Since ZFS is a new technology for Linux, it can be helpful to understand some of its benefits, particularly in comparison to BTRFS, another popular next-generation Linux filesystem:
 
* On Linux, the ZFS code can be updated independently of the kernel to obtain the latest fixes. btrfs is exclusive to Linux and you need to build the latest kernel sources to get the latest fixes.
 
* ZFS is supported on multiple platforms. The platforms with the best support are Solaris, FreeBSD and Linux. Other platforms with varying degrees of support are NetBSD, Mac OS X and Windows. btrfs is exclusive to Linux.
 
* ZFS has the Adaptive Replacement Cache replacement algorithm while btrfs uses the Linux kernel's Last Recently Used replacement algorithm. The former often has an overwhelmingly superior hit rate, which means fewer disk accesses.
 
* ZFS has the ZFS Intent Log and SLOG devices, which accelerates small synchronous write performance.
 
* ZFS handles internal fragmentation gracefully, such that you can fill it until 100%. Internal fragmentation in btrfs can make btrfs think it is full at 10%. Btrfs has no automatic rebalancing code, so it requires a manual rebalance to correct it.
 
* ZFS has raidz, which is like RAID 5/6 (or a hypothetical RAID 7 that supports 3 parity disks), except it does not suffer from the RAID write hole issue thanks to its use of CoW and a variable stripe size. btrfs gained integrated RAID 5/6 functionality in Linux 3.9. However, its implementation uses a stripe cache that can only partially mitigate the effect of the RAID write hole.
 
* ZFS send/receive implementation supports incremental update when doing backups. btrfs' send/receive implementation requires sending the entire snapshot.
 
* ZFS supports data deduplication, which is a memory hog and only works well for specialized workloads. btrfs has no equivalent.
 
* ZFS datasets have a hierarchical namespace while btrfs subvolumes have a flat namespace.
 
* ZFS has the ability to create virtual block devices called zvols in its namespace. btrfs has no equivalent and must rely on the loop device for this functionality, which is cumbersome.
 
The only area where btrfs is ahead of ZFS is in the area of small file
efficiency. btrfs supports a feature called block suballocation, which
enables it to store small files far more efficiently than ZFS. It is
possible to use another filesystem (e.g. reiserfs) on top of a ZFS zvol
to obtain similar benefits (with arguably better data integrity) when
dealing with many small files (e.g. the portage tree).
 
For a quick tour of ZFS and have a big picture of its common operations you can consult the page [[ZFS Fun]].
 
=== Disclaimers ===
 
{{fancywarning|This guide is a work in progress. Expect some quirks.}}
{{fancyimportant|'''Since ZFS was really designed for 64 bit systems, we are only recommending and supporting 64 bit platforms and installations. We will not be supporting 32 bit platforms'''!}}
== Downloading the ISO (With ZFS) ==
In order for us to install Funtoo on ZFS, you will need an environment that already provides the ZFS tools. Therefore we will download a customized version of System Rescue CD with ZFS included.  


<pre>
== Prerequisites ==
Name: sysresccd-4.2.0_zfs_0.6.2.iso  (545 MB)
Release Date: 2014-02-25
md5sum 01f4e6929247d54db77ab7be4d156d85
</pre>


{{fancyimportant|''ZFS is designed for 64-bit systems. We only recommend and support 64-bit platforms and installations!''}}
{{fancywarning|''ZFS v07.5 (latest in ports as of 18 Jan 2018) is compatible with kernel versions 2.6.32 - 4.14''}}
{{fancywarning|''The guide is under rewrite''}}
It is recommended to give the entire disk to ZFS. As such, this guide will only show how to install ZFS on the whole disk, using legacy boot. Installing on UEFI requires a separate partition for /boot, formatted as FAT32, and is out of the scope of this guide, even though installation on UEFI is certainly possible. Also, this guide will not cover anything related to encryption (native encryption is available: https://github.com/zfsonlinux/zfs/commit/b52563034230b35f0562b6f40ad1a00f02bd9a05).


'''[http://ftp.osuosl.org/pub/funtoo/distfiles/sysresccd/ Download System Rescue CD with ZFS]'''<br />
=== Downloading the ISO (With ZFS) ===
In order to install Funtoo on ZFS, you will need an environment, such as live media, that provides the ZFS tools. This guide will utilize the Ubuntu Desktop 16.04 (live) DVD for amd64.
* Download from '''[https://build.funtoo.org/distfiles/ubuntu-17.10.1-desktop-amd64.iso]'''


== Creating a bootable USB from ISO (From a Linux Environment) ==
== Creating a bootable USB from ISO (From a Linux Environment) ==
After you download the iso, you can do the following steps to create a bootable USB:
After you download the iso, you can do the following steps to create a bootable USB:
{{note|The size of the iso is approximately 1.5 GB.}}
Insert your blank USB media into a USB port. Then, inspect the kernel ring buffer with {{c|dmesg}} to identify the device name of your USB storage.
{{console|title=dmesg|body=
[  +5.533491] usb 6-2: new SuperSpeed USB device number 4 using xhci_hcd
[  +0.022995] usb 6-2: New USB device found, idVendor=1b1c, idProduct=1a0c
[  +0.000006] usb 6-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  +0.000003] usb 6-2: Product: Voyager Mini 3.0
[  +0.000003] usb 6-2: Manufacturer: Corsair
[  +0.000002] usb 6-2: SerialNumber: 0123456789ABCDEF
[  +0.001095] usb-storage 6-2:1.0: USB Mass Storage device detected
[  +0.000080] scsi host15: usb-storage 6-2:1.0
[  +1.000772] scsi 15:0:0:0: Direct-Access    Corsair  Voyager Mini 3.0 PMAP PQ: 0 ANSI: 6
[  +0.000615] sd 15:0:0:0: Attached scsi generic sg6 type 0
[  +0.000110] sd 15:0:0:0: [sdg] 60566016 512-byte logical blocks: (31.0 GB/28.9 GiB)
[  +0.000209] sd 15:0:0:0: [sdg] Write Protect is off
[  +0.000004] sd 15:0:0:0: [sdg] Mode Sense: 2b 00 00 08
[  +0.000227] sd 15:0:0:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  +0.359532] sd 15:0:0:0: [sdg] Attached SCSI removable disk
}}
In this example, {{c|[sdg]}} indicates that the device is {{c|/dev/sdg}}.


<console>
A quick and easy way to create a bootable USB is to write the ISO data to the USB device using {{c|dd}}.
Make a temporary directory
{{console|body=
# ##i##mkdir /tmp/loop
###i## dd if=/path/to/iso/ubuntu-16.04-desktop-amd64.iso of=/dev/sdg bs=4K
 
}}
Mount the iso
# ##i##mount -o ro,loop /root/sysresccd-4.2.0_zfs_0.6.2.iso /tmp/loop
 
Run the usb installer
# ##i##/tmp/loop/usb_inst.sh
</console>


That should be all you need to do to get your flash drive working.
Once this has completed, remove and use this USB to boot the target system that will receive Funtoo Linux.


== Booting the ISO ==
== Booting the ISO ==
Using legacy (BIOS) boot mode, boot the ISO and allow Ubuntu to load the graphical environment. You will be presented with a "Welcome" dialog (titled ''Install (as superuser)''). Select the option "Try Ubuntu".


{{fancywarning|'''When booting into the ISO, Make sure that you select the "Alternate 64 bit kernel (altker64)". The ZFS modules have been built specifically for this kernel rather than the standard kernel. If you select a different kernel, you will get a fail to load module stack error message.'''}}
Once the desktop has loaded, open the search bar by left-clicking on the top-left icon ("Search your computer"). Pressing Alt+F1 should also open this search bar. Search for and open the Terminal application.
 
== Creating partitions ==
There are two ways to partition your disk: You can use your entire drive and let ZFS automatically partition it for you, or you can do it manually.


We will be showing you how to partition it '''manually''' because if you partition it manually you get to create your own layout, you get to have your own separate /boot partition (Which is nice since not every bootloader supports booting from ZFS pools), and you get to boot into RAID10, RAID5 (RAIDZ) pools and any other layouts due to you having a separate /boot partition.
In the terminal, issue the following commands to install and load the required ZFS module.


==== gdisk (GPT Style) ====
{{console|body=
ubuntu@ubuntu:~$##i## sudo -i
root@ubuntu:~###i## apt-add-repository universe
root@ubuntu:~###i## apt update
root@ubuntu:~###i## apt install --yes debootstrap gdisk zfs-initramfs
}}


'''A Fresh Start''':
Verify that the ZFS kernel module has loaded.
{{console|body=
root@ubuntu:~# ##i## dmesg {{!}} grep ZFS
[  377.595348] ##r##ZFS##!r##: Loaded module v0.6.5.6-0ubuntu10, ##r##ZFS##!r## pool version 5000, ##r##ZFS##!r## filesystem version 5
}}


First lets make sure that the disk is completely wiped from any previous disk labels and partitions.
== Drive Partitions ==
We will also assume that <tt>/dev/sda</tt> is the target drive.<br />
We are letting ZFS automatically partition the drive. This is ideal for our example using a single disk and legacy (BIOS) boot.


=== Creating of pool ===
Create a ZFS Storage Pool using a single whole disk.
<console>
<console>
# ##i##sgdisk -Z /dev/sda
root@ubuntu:~###i## zpool create -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o feature@spacemap_histogram=enabled -o feature@enabled_txg=enabled -o feature@extensible_dataset=enabled -o feature@bookmarks=enabled -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -O atime=off -O xattr=sa -O compression=lz4 -m none -R /mnt/funtoo rpool /dev/disk/by-id/foo
</console>
</console>
The options used here are
{{TableStart}}
{{2ColHead|option|description}}
{{2Col|{{c|create}}|Use {{c|zpool}} to create a ZFS Storage Pool.}}
{{2Col|{{c|-d}}|Required. Disables enabling all availabe zfs features. This is needed as the GRUB bootloader currently can't handle all advanced zfs features, not specifying this flag on pool creation (or running 'zpool upgrade' on the pool at a later point in time) will make the pool incompatible with GRUB.}}
{{2Col|{{c|(multiple) -o feature@<name>{{=}}enabled}}|Explicitely enable all features currently supported by GRUB. See 'man zpool-features' for details on their effects.}}
{{2Col|{{c|-f}}|Force the use of the selected disk.}}
{{2Col|{{c|-o ashift{{=}}12}}|Alignment of the pool to underlying hard drive sectors. The recommended value is 12, which corresponds to 2^12 Bytes or 4 KiB. This value is typical for present-day HDD's. Can only be set once at pool creation.}}
{{2Col|{{c|-o cachefile{{=}}/tmp/zpool.cache}}|Create a pool configuration cache and place it in {{f|/tmp}}. This will be required for our Funtoo install.}}
{{2Col|{{c|-O normalization{{=}}formD}}|Recommended. Set the default Unicode (UTF-8) normalization for future filesystems (created within this pool) to 'formD'.}}
{{2Col|{{c|-O atime{{=}}off}}|Recommended. As a default preference, set future filesystems (created within this pool) to not update file access time. Useful if we want to reduce writes to disk (e.g., Solid State Drives). Can cause problems for mailers and other software that rely on file access-time data.}}
{{2Col|{{c|-O xattr{{=}}sa}}|Recommended. As a default preference, set future filesystems (created within this pool) to store extended file attributes in a more efficient manner.}}
{{2Col|{{c|-O compression{{=}}lz4}}|Recommended. As a default preference, set future filesystems (created within this pool) to be compressed using the lz4 algorithm. Useful in general as the runtime cost for compression is neglible on these days while on-disk space still costs money.}}
{{2Col|{{c|-m none}}|Do not set mountpoint for this storage pool. (This guide will address this later).}}
{{2Col|{{c|-R /mnt/funtoo}}|Alternate root directory. Essentially a temporary 'mount point' for our pool.}}
{{2Col|{{c|rpool}}|The name of this ZFS Storage Pool. The pool name is irrelevant. {{c|rpool}} will be used throughout this guide.}}
{{2Col|{{c|/dev/disk/by-id/foo}}|The path to the physical disk. e.g. /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF. Known in 'ZFS-speak' as a 'VDEV'.}}
{{TableEnd}}
Please note that the options 'ashift' and 'cachefile' are preceded with a lowercase '-o', while 'normalization' and 'atime' are preceded with an uppercase '-O'.


{{fancywarning|This is a destructive operation and the program will not ask you for confirmation! Make sure you really don't want anything on this disk.}}
Without additional options, this will create our storage pool and enable all 'features' available under version 0.6.5.6. The pool will be automatically mounted at the (temporary) location {{f|/mnt/funtoo}}.


Now that we have a clean drive, we will create the new layout.
To confirm the presence of our newly created pool:
{{console|body=
root@ubuntu:~###i## zpool status
  pool: rpool
state: ONLINE
  scan: none requested
config:


First open up the application:
NAME                                            STATE    READ WRITE CKSUM
rpool                                            ONLINE      0    0    0
  ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF  ONLINE      0    0    0


<console>
errors: No known data errors
# ##i##gdisk /dev/sda
}}
</console>


'''Create Partition 1''' (boot):
=== Create ZFS Datasets ===
Now, we will create one or more ZFS datasets within our storage pool. These will contain Funtoo Linux.
==== Create the Root file system (Required) ====
<console>
<console>
Command: ##i##n ↵
# ##i## zfs create -o mountpoint=none -o canmount=off rpool/ROOT
Partition Number: ##i##↵
# ##i## zfs create -o mountpoint=/ rpool/ROOT/funtoo
First sector: ##i##↵
Last sector: ##i##+250M ↵
Hex Code: ##i##↵
</console>
 
'''Create Partition 2''' (BIOS Boot Partition):
<console>Command: ##i##n ↵
Partition Number: ##i##↵
First sector: ##i##↵
Last sector: ##i##+32M ↵
Hex Code: ##i##EF02 ↵
</console>
 
'''Create Partition 3''' (ZFS):
<console>Command: ##i##n ↵
Partition Number: ##i##↵
First sector: ##i##↵
Last sector: ##i##↵
Hex Code: ##i##bf00 ↵
 
Command: ##i##p ↵
 
Number  Start (sector)    End (sector)  Size      Code  Name
  1            2048          514047  250.0 MiB  8300  Linux filesystem
  2          514048          579583  32.0 MiB    EF02  BIOS boot partition
  3          579584      1953525134  931.2 GiB  BF00  Solaris root
 
Command: ##i##w ↵
</console>
</console>
 
==== Create optional Datasets====
 
The following optional datasets are provided as examples. It is up to the user to create their own datasets.
=== Format your /boot partition ===
===== Home=====
 
<console>
<console>
# ##i##mkfs.ext2 -m 1 /dev/sda1
# ##i## zfs create -o mountpoint=/home rpool/HOME
</console>
</console>
 
=====Build directory=====
=== Create the zpool ===
We will first create the pool. The pool will be named `tank` and the disk will be aligned to 4096 (using ashift=12)
<console># ##i##zpool create -f -o ashift=12 -o cachefile= -O compression=on -m none -R /mnt/funtoo tank /dev/sda3</console>
 
=== Create the zfs datasets ===
We will now create some datasets. For this installation, we will create a small but future proof amount of datasets. We will have a dataset for the OS (/), and your swap. We will also show you how to create some optional datasets: <tt>/home</tt>, <tt>/var</tt>, <tt>/usr/src</tt>, and <tt>/usr/portage</tt>.
 
<console>
<console>
Create some empty containers for organization purposes, and make the dataset that will hold /
# ##i## zfs create -o mountpoint=none -o canmount=off rpool/FUNTOO
# ##i##zfs create -p tank/funtoo
# ##i## zfs create -o mountpoint=/var/tmp/portage -o compression=lz4 -o sync=disabled rpool/FUNTOO/build
# ##i##zfs create -o mountpoint=/ tank/funtoo/root
 
Optional, but recommended datasets: /home
# ##i##zfs create -o mountpoint=/home tank/funtoo/home
 
Optional datasets: /usr/src, /usr/portage/{distfiles,packages}
# ##i##zfs create -o mountpoint=/usr/src tank/funtoo/src
# ##i##zfs create -o mountpoint=/usr/portage -o compression=off tank/funtoo/portage
# ##i##zfs create -o mountpoint=/usr/portage/distfiles tank/funtoo/portage/distfiles
# ##i##zfs create -o mountpoint=/usr/portage/packages tank/funtoo/portage/packages
</console>
</console>
=====Swap on ZFS=====
With some careful tuning a swap partition can be created on a ZFS 'volume' [https://github.com/zfsonlinux/zfs/wiki/FAQ]. For a 2 GB swapfs:
{{console|body=
# ##i## zfs create -V 2G -b $(getconf PAGESIZE) -o logbias=throughput -o sync=always -o primarycache=metadata rpool/swap
# ##i## mkswap /dev/zvol/rpool/swap
# ##i## swapon /dev/zvol/rpool/swap
}}
Note that swap on ZFS has known stability issues. If the user has decided to use swap, please take following into account:
Always use long <code>/dev/zvol</code> aliases in configuration files. Never use a short <code>/dev/zdX</code> device name.


=== Create your swap zvol ===
To confirm the presence of the filesystems that we have created:
For modern machines that have greater than 4 GB of RAM, A swap size of 2G should be enough. However if your machine doesn't have a lot of RAM, the rule of thumb is either 2x the RAM or RAM + 1 GB.
{{console|body=
root@ubuntu:~# ##i##zfs list -t all
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool              660K  19.3G    96K  none
rpool/HOME          96K  19.3G    96K  /mnt/funtoo/home
rpool/ROOT          192K  19.3G    96K  none
rpool/ROOT/funtoo    96K  19.3G    96K  /mnt/funtoo
}}


For this tutorial we will assume that it is a newer machine and make a 2 GB swap.
===Make the root filesystem bootable===
 
{{important|Do not skip this!}}
<console>
When booting from ZFS, you must specify a boot device and a root file system within the pool that was identified by the boot device. By default, the dataset selected for booting is the one identified by the pool's <code>bootfs</code> property.
# ##i##zfs create -o sync=always -o primarycache=metadata -o secondarycache=none -o volblocksize=4K -V 2G tank/swap
</console>
{{fancywarning|ZFS swap is not stable and should be used with precautions.}}


=== Format your swap zvol ===
<console>
<console>
# ##i##mkswap -f /dev/zvol/tank/swap
# ##i##zpool set bootfs=rpool/ROOT/funtoo rpool
# ##i##swapon /dev/zvol/tank/swap
</console>
</console>
Now we will continue to install funtoo.


== Installing Funtoo ==
== Installing Funtoo ==
Now that the initial ZFS pool and datasets have been created, we can unpack the stage 3 tarball and proceed with basic system installation. This part does not differ much from a regular Funtoo Linux installation (https://www.funtoo.org/Install#Installing_the_Stage_3_tarball). We recommend using <code>funtoo-current</code> stages for ZFS.


=== Pre-Chroot ===
{{console|body=
# ##i##cd /mnt/funtoo
# ##i##wget https://build.funtoo.org/next/x86-64bit/generic_64/stage3-latest.tar.xz
}}
Extract the contents with the following command. If the user is using an arch-optimized stage 3 tarball, substitute in the actual filename.


<console>
{{console|body=
Go into the directory that you will chroot into
# ##i##cd /mnt/funtoo
# ##i##cd /mnt/funtoo
# ##i##tar xpf stage3-latest.tar.xz
}}


Make a boot folder and mount your boot drive
Now, we need to create a chroot environment:
# ##i##mkdir boot
# ##i##mount /dev/sda1 boot
</console>


[[Funtoo_Linux_Installation|Now download and extract the Funtoo stage3 ...]]
{{console|body=
 
# ##i##cd /mnt/funtoo
Once you've extracted the stage3, do a few more preparations and chroot into your new funtoo environment:
 
<console>
Bind the kernel related directories
# ##i##mount -t proc none proc
# ##i##mount -t proc none proc
# ##i##mount --rbind /sys sys
# ##i##mount --rbind /dev dev
# ##i##mount --rbind /dev dev
# ##i##mount --rbind /sys sys
}}


Copy network settings
An important step is to copy the ZFS cache into the chroot. The ZFS cache was created when <code>rpool</code> was created with the <code>-o cachefile=/tmp/zpool.cache</code> flag.
# ##i##cp -f /etc/resolv.conf etc
{{console|body=
# ##i##mkdir -p /mnt/funtoo/etc/zfs
# ##i##cp /tmp/zpool.cache /mnt/funtoo/etc/zfs/zpool.cache
}}


Make the zfs folder in 'etc' and copy your zpool.cache
You will also want to copy over {{f|resolv.conf}} in order to have proper resolution of Internet hostnames from inside the chroot:
# ##i##mkdir etc/zfs
# ##i##cp /etc/zfs/zpool.cache etc/zfs


Chroot into Funtoo
{{console|body=
# ##i##env -i HOME=/root TERM=$TERM chroot . bash -l
# ##i##cp /etc/resolv.conf /mnt/funtoo/etc/
</console>
}}
We are now ready to <code>chroot</code>.


=== In Chroot ===
{{console|body=
# ##i##chroot /mnt/funtoo /bin/bash
# ##i##export PS1="(chroot) $PS1"; cd
}}


<console>
==Configuring your system==
Create a symbolic link to your mountpoints
Configure your system according to the main [[Install|install guide]]. During startup, ZFS filesystems will be mounted '''without''' needing any entries in {{f|/etc/fstab}}. Comment out all entries in {{f|/etc/fstab}} except for partitions such as CD-ROMs, tmpfs, etc., if used.
# ##i##ln -sf /proc/mounts /etc/mtab


Sync your tree
If you created a '''swap''' volume earlier, add an appropriate entry to {{f|/etc/fstab}}.
# ##i##emerge --sync
{{console|body=
</console>
# ##i## echo /dev/zvol/rpool/swap none swap defaults 0 0 >> /etc/fstab
}}
Note that swap on ZFS is unstable. If you decide to use swap, please take following into account:
always use long <code>/dev/zvol</code> aliases in configuration files. Never use a short <code>/dev/zdX</code> device name.


=== Add filesystems to /etc/fstab ===
Next, update the portage tree:
{{console|body=
###i## ego sync
###i## env-update
###i## source /etc/profile
}}


Before we continue to compile and or install our kernel in the next step, we will edit the <tt>/etc/fstab</tt> file because if we decide to install our kernel through portage, portage will need to know where our <tt>/boot</tt> is, so that it can place the files in there.
== Installing ZFS userspace and bootloader==
=== Installing the ZFS userspace tools and kernel modules ===


Edit <tt>/etc/fstab</tt>:
Install the ZFS packages and sync the portage tree, if required:
{{console|body=
###i## ego sync
###i## emerge --ask sys-fs/zfs
}}


<pre>
Once it has successfully merged, add the following services to the boot runlevel of OpenRC:
# <fs>                  <mountpoint>    <type>          <opts>          <dump/pass>
{{console|body=
###i## rc-update add zfs-import boot
###i## rc-update add zfs-mount boot
}}
Add another two services to the default runlevel:
{{console|body=
###i## rc-update add zfs-share default
###i## rc-update add zfs-zed default
}}


/dev/sda1              /boot          ext2            defaults        0 2
=== Create a ZFS-friendly initramfs ===
/dev/zvol/tank/swap    none            swap            sw              0 0
The Funtoo stage3 includes a linux kernel and initramfs. The initramfs is designed to mount and start Funtoo Linux on a variety of file systems. The initramfs contained within the stage3 will '''not''' mount and start Funtoo in our ZFS storage pool. We must create an updated 'ZFS-friendly' initramfs.
</pre>


== Kernel Configuration ==
Optional: Update to the latest {{package|sys-kernel/genkernel}}:
...wip
{{console|body=
# ##i##emerge --oneshot sys-kernel/genkernel
}}


== Installing the ZFS userspace tools and kernel modules ==
Use {{c|genkernel}} to create an initramfs capable of mounting our ZFS Storage Pool via the {{c|--zfs}} switch. Adjust {{c|--makeopts}} according to the number of available threads:
Emerge {{Package|sys-fs/zfs}}. This package will bring in {{Package|sys-kernel/spl}}, and {{Package|sys-fs/zfs-kmod}} as its dependencies:
{{console|body=
# ##i##genkernel initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs
##g##* Funtoo Linux Genkernel; Version 3.4.40.11-funtoo##!g##
##g##*##!g## Running with options: initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs


<console>
##g##*##!g## Linux Kernel 4.5.2-1 for x86_64...
# ##i##emerge zfs
##g##*##!g## .. with config file /usr/src/linux-debian-sources-4.5.2/.config
</console>
##g##*##!g## busybox: >> Using cache
##g##*##!g## initramfs: >> Initializing...
##g##*##!g##        >> Appending base_layout cpio data...
##g##*##!g##        >> Appending auxilary cpio data...
##g##*##!g##        >> Copying keymaps
##g##*##!g##        >> Appending busybox cpio data...
##g##*##!g##        >> Appending modules cpio data...
##g##*##!g##         >> Appending zfs cpio data...
cp: cannot stat ‘/etc/zfs/zdev.conf’: No such file or directory
##y##*##!y## Could not copy file /etc/zfs/zdev.conf for ZFS
##g##*##!g##        >> Appending blkid cpio data...
##g##*##!g##        >> Appending modprobed cpio data...
##g##*##!g##        >> Compressing cpio data (.xz)...


Check to make sure that the zfs tools are working. The <code>zpool.cache</code> file that you copied before should be displayed.
##y##*##!y## WARNING... WARNING... WARNING...
##y##*##!y## Additional kernel cmdline arguments that *may* be required to boot properly...
##y##*##!y## add "dozfs" for ZFS volume management support
##y##*##!y## add either "real_root=ZFS" (bootfs autodetection) or "real_root=ZFS=<dataset>" to boot from a ZFS dataset


<console>
##g##*##!g## Do NOT report kernel bugs as genkernel bugs unless your bug
# ##i##zpool status
##g##*##!g## is about the default genkernel configuration...
# ##i##zfs list
##g##*##!g##
</console>
##g##*##!g## Make sure you have the latest ~arch genkernel before reporting bugs.
}}


If everything worked, continue.
Confirm the presence of the new initramfs:
{{console|body=
###i## ls /boot/*genkernel*
/boot/initramfs-genkernel-x86_64-4.5.2-1
}}


== Create the initramfs ==
Grub expects the initramfs filename to be of the form: <code>initramfs-${KNAME}-${ARCH}-${KV}</code>. This guide assumes the usage of debian-sources. Tailor it to your specific kernel name, platform, and kernel version. Forgetting to rename <code>initramfs-genkernel</code> to <code>initramfs-debian-sources</code> will render the system unbootable.
There are two ways to do this, you can use "genkernel" or "bliss-initramfs". Both will be shown.
{{console|body=
###i## cd /boot
###i## mv initramfs-genkernel-x86_64-4.5.2-1 initramfs-debian-sources-x86_64-4.5.2-1
###i## ls /boot/*initramfs*
/boot/initramfs-debian-sources-x86_64-4.5.2-1
}}


=== genkernel ===
=== Installing GRUB 2  ===
Install genkernel and run it:
GRUB 2 must be built with support for ZFS Storage Pools on a single disk. This is achieved using the 'libzfs' USE flag.
<console>
<console>
# ##i##emerge genkernel
# ##i##echo "sys-boot/grub libzfs" >> /etc/portage/package.use
</console>


You only need to add --luks if you used encryption
{{Note|If you have defined GRUB_PLATFORMS in your {{f|/etc/portage/make.conf}}, please ensure that it includes 'pc': e.g. GRUB_PLATFORMS{{=}}"efi-64 pc".
# ##i##genkernel --zfs --luks initramfs
</console>


== Installing & Configuring the Bootloader ==
This is required for booting in BIOS (non-UEFI) mode, as described in this guide.}}


=== GRUB 2  ===
Now, install GRUB:
<console>
<console>
# ##i##emerge grub
# ##i##emerge grub
</console>
</console>


You can check that grub is version 2.00 by typing the following command:
=== Configuring the Bootloader ===
When {{c|zpool}} created our storage pool (rpool), it created partitions under a GPT scheme. In order to boot Funtoo Linux on a GPT partion under legacy (BIOS) boot, <code>sys-boot/grub</code> requires a small partition, called a BIOS boot partition. By design, ZFS ({{c|zpool}}) left a very small unpartitioned space at the beginning of the disk. We will use <code>sgdisk</code>, which is part of <code>sys-apps/gptfdisk</code>, to format this free space into a BIOS boot partition.


<console>
{{console|body=
# ##i##grub-install --version
###i## sgdisk -a1 -n2:48:2047 -t2:EF02 -c2:"BIOS boot partition" /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF
grub-install (GRUB) 2.00
}}
</console>


Now install grub to the drive itself (not a partition):
To avoid problems with GRUB, use {{c|partx}} to refresh the list of partitions that are 'seen' by the kernel. Do this for each drive (/dev/sda, /dev/sdb, etc.).
<console>
{{console|body=
# ##i##grub-install /dev/sda
###i## partx -u /dev/sda
</console>
}}


You should receive the following message:
A quick check to verify that GRUB 2 sees/supports ZFS:
{{console|body=
###i## touch /etc/mtab
###i## grub-probe /
zfs
}}


<console>
Installing GRUB2 to disk is as easy as:
{{console|body=
###i## grub-install /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF
Installing for i386-pc platform.
Installation finished. No error reported.
Installation finished. No error reported.
</console>
}}


You should now see some a grub directory with some files inside your /boot folder:
Now, it's time for us to create grub's configuration file. First, we must edit a few GRUB 2 settings in {{f|/etc/default/grub}}:
{{important|The following is required to boot Funtoo Linux on ZFS!}}
Replace the line
{{file|name=/etc/default/grub|desc=before|body=
#GRUB_CMDLINE_LINUX=""
}}
with
{{file|name=/etc/default/grub|desc=after|body=
GRUB_CMDLINE_LINUX="dozfs real_root=ZFS=rpool/ROOT/funtoo"
}}


<console>
Now, create GRUB 2 configuration file:
# ##i##ls -l /boot/grub
{{console|body=
total 2520
###i## grub-mkconfig -o /boot/grub/grub.cfg
-rw-r--r-- 1 root root    1024 Jan  4 16:09 grubenv
Generating grub configuration file ...
drwxr-xr-x 2 root root    8192 Jan 12 14:29 i386-pc
Found linux image: /boot/kernel-debian-sources-x86_64-4.5.2-1
drwxr-xr-x 2 root root    4096 Jan 12 14:28 locale
Found initrd image: /boot/initramfs-genkernel-x86_64-4.5.2-1
-rw-r--r-- 1 root root 2555597 Feb  4 11:50 unifont.pf2
done
</console>
}}


=== boot-update ===
If grub-mkconfig cannot find the initrd image, manually add initrd to the generated {{c|/boot/grub/grub.cfg}}, below the linux line:
boot-update comes as a dependency of grub2, so if you already installed grub, it's already on your system!
{{file|name=/boot/grub/grub.cfg|desc=adding initrd below the linux line|body=
echo    'Loading Linux x86_64-4.5.2-1 ...'
linux  /ROOT/funtoo@/boot/kernel-debian-sources-x86_64-4.5.2-1 root=ZFS=rpool/ROOT/funtoo ro dozfs real_root=ZFS
echo    'Loading initial ramdisk ...'
initrd  /ROOT/funtoo@/boot/initramfs-debian-sources-x86_64-4.5.2-1
}}


==== Genkernel ====
A quick check to verify that the GRUB ZFS module is in place:
If your using genkernel you must add 'real_root=ZFS=<root>' and 'dozfs' to your params.
{{console|body=
Example entry for <tt>/etc/boot.conf</tt>:
###i## ls /boot/grub/*/zfs.mod
/boot/grub/i386-pc/zfs.mod
}}


<pre>
== Final configuration ==
"Funtoo ZFS" {
Configure your network according to the main [[Install#Configuring_your_network|installation guide]].
        kernel vmlinuz[-v]
        initrd initramfs-genkernel-x86_64[-v]
        params real_root=ZFS=tank/funtoo/root
        params += dozfs=force
        # Also add 'params += crypt_root=/dev/sda3' if you used encryption
        # Adjust the above setting to your system if needed


        # You should also add 'root=none' to your params (not 'params +=') if you plan to use it along with boot-update
Set the root password.
        # If root variable will not be set, boot-update will fail to generate boot.conf
{{console|body=
        # This is right for <=sys-boot/boot-update-1.6.11 on 16.08.2014 date
(chroot) ###i## passwd
}
}}
</pre>


==== Bliss Initramfs Creator ====
Exit the chroot and export your ZFS storage pool.
If you used Bliss Initramfs Creator then all you need to do is add 'root=<root>' to your params.
{{console|body=
Example entry for <tt>/etc/boot.conf</tt>:
(chroot) ###i## exit
###i## umount -lR {dev,proc,sys}
###i## cd /
###i## zpool export rpool
}}


<pre>
Restart to boot into Funtoo Linux on a ZFS root!
"Funtoo ZFS" {
        kernel vmlinuz[-v]
        initrd initrd[-v]
        params root=tank/funtoo/root quiet
        # If you have an encrypted device with a regular passphrase,
        # you can add the following line
        params += enc_root=/dev/sda3 enc_type=pass
}
</pre>


After editing /etc/boot.conf, you just need to run boot-update to update grub.cfg
== After reboot ==
===Snapshot===
Take a snapshot of your Funtoo ''at installation''.
{{console|body=
###i## zfs snapshot rpool/ROOT/funtoo@install
}}
The use of snapshots, including sending snapshots as a method of backup, are not covered in this guide. See [[ZFS_Install_Guide#Further_Reading|Further Reading]].


<console>
===ZFS Adjustable Replacement Cache (ARC) size===
###i## boot-update
The Adjustable Replacement Cache (ARC) is a fundamental part of ZFS. Refer to this [https://pthree.org/2012/12/07/zfs-administration-part-iv-the-adjustable-replacement-cache%20link article] by Aaron Toponce for details on how to set up an ARC.
</console>


=== bliss-boot ===
Without configuration, ZFS will use up to 50% of your memory (RAM) for the ARC. It is possible to change this maximum. There are different ways to achieve this on both a temporary and persistent basis. One such way is to create and edit the file {{f|/etc/modprobe.d/zfs.conf}}, which affects the ZFS kernel module.
This is a new program that is designed to generate a simple, human-readable/editable, configuration file for a variety of bootloaders. It currently supports grub2, extlinux, and lilo.


You can install it via the following command:
{{file|name=/etc/modprobe.d/zfs.conf|desc=set maximum ARC size to 4 GiB|body=
<console>
options zfs zfs_arc_max=4294967296
# ##i##emerge bliss-boot
}}
</console>
where {{c|zfs_arc_max}} is set to a value in Bytes. After configuring this file, re-generate the initramfs. Rebooting will then apply this change.


==== Bootloader Configuration ====
To apply this change immediately without a reboot, issue the command:
In order to generate our bootloader configuration file, we will first configure bliss-boot so that it knows what we want. The 'bliss-boot' configuration file is located in '''/etc/bliss-boot/conf.py'''. Open that file and make sure that the following variables are set appropriately:
{{console|body=
# ##i## echo 4294967296 >> /sys/module/zfs/parameters/zfs_arc_max
}}


<pre>
Reference: https://wiki.gentoo.org/wiki/ZFS#ARC
# This should be set to the bootloader you installed earlier: (grub2, extlinux, and lilo are the available options)
bootloader = "grub2"


# This should be set to the kernel you installed earlier
===After Kernel or ZFS updates===
default = "3.12.13-KS.02"
The default Funtoo Linux kernel ({{package|sys-kernel/debian-sources}}) does not automatically build a ZFS-capable initramfs. Similarly, the package {{c|sys-fs/zfs}} and its dependencies do not automatically build a ZFS-capable initramfs.
</pre>


Scroll all the way down until you find 'kernels'. You will need to add the kernels and the options
After each kernel update, you must recreate an initramfs. Similarly, {{c|sys-fs/zfs}} updates also require you to regenerate an initramfs. This is especially true for updates to {{c|sys-fs/zfs}} that introduce new storage pool features. Neglecting to update your initramfs will make your system unbootable.
you want for these kernels here. Below are a few configuration options depending if you are using
'''bliss-initramfs''' or '''genkernel'''.


===== Genkernel =====
First, rebuild {{c|sys-fs/zfs}}, {{c|sys-fs/zfs-kmod}} and {{c|sys-kernel/spl}}:


<pre>
{{console|body=
kernel = {
# ##i##emerge --ask --verbose -1 zfs zfs-kmod spl
    '3.12.13-KS.02' : 'real_root=ZFS=tank/funtoo/root dozfs=force quiet',
}}
}
</pre>


'''If you are using encryption you can add the crypt_root option:'''
Then, follow the earlier instructions to create a new initramfs:
{{console|body=
# ##i##genkernel initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs
# ##i##cd /boot
# ##i##mv initramfs-genkernel-x86_64-4.5.2-1 initramfs-debian-sources-x86_64-4.5.2-1
}}


<pre>
Finally, repeat the earlier instructions for generating {{c|/boot/grub/grub.cfg}} and manually adding the new initramfs as initrd to the generated {{c|/boot/grub/grub.cfg}}.
kernel = {
    '3.12.13-KS.02' : 'real_root=ZFS=tank/funtoo/root dozfs=force crypt_root=/dev/sda3 quiet',
}
</pre>


===== Bliss Initramfs Creator =====
== Further Reading ==
<pre>
ZFS has many interesting features not covered by this guide.
kernel = {
    '3.12.13-KS.02' : 'root=tank/funtoo/root quiet',
}
</pre>


'''If you are using encryption then you would let the initramfs know:'''
Useful information and instructions can be found in the online reference manuals. See {{c|man zpool}} and {{c|man zfs}}.


#"What type of encryption authentication you want to use? ('''enc_type=''')
Aaron Toponce's Zpool/ZFS Administration Guides - https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs
::* pass = will ask for passphrase directly
::* key = a plain unencrypted key file
::* key_gpg = an encrypted key file
#"Where is the encrypted drive?" ('''enc_root=''')
#"Where is the root pool after it has been decrypted?" ('''root=''')


<pre>
ZFS on Linux - http://www.zfsonlinux.org
kernel = {
    '3.12.13-KS.02' : 'root=tank/funtoo/root enc_root=/dev/sda3 enc_type=pass quiet',
}
</pre>


==== Generate the configuration ====
ZFS - Gentoo Wiki - https://wiki.gentoo.org/wiki/ZFS
Now that we have configure our '''/etc/bliss-boot/conf.py''' file, we can generate our config. Simply run the following command:
 
<console>
# ##i##bliss-boot
</console>
 
This will generate a configuration file for the bootloader you specified previously in your current directory. You can check your config file before hand to make sure it doesn't have any errors. Simply open either: grub.cfg, extlinux.conf, or lilo.conf.
 
Once you have checked it for errors, place this file in the correct directory:
 
* grub2 = /boot/grub/
* extlinux = /boot/extlinux/
* lilo = /etc/lilo.conf
 
=== LILO (Optional if you are using another bootloader) ===
Now that bliss-boot generated the lilo.conf file, move that config file to its appropriate location
and install lilo to the MBR:
 
<console>
# ##i##mv lilo.conf /etc
# ##i##lilo
 
You should see the following:
 
Warning: LBA32 addressing assumed
Added Funtoo + *
One warning was issued
</console>
 
== Final configuration ==
=== Add the zfs tools to openrc ===
<console># ##i##rc-update add zfs boot</console>
 
=== Clean up and reboot ===
We are almost done, we are just going to clean up, '''set our root password''', and unmount whatever we mounted and get out.
 
<console>
Delete the stage3 tarball that you downloaded earlier so it doesn't take up space.
# ##i##cd /
# ##i##rm stage3-latest.tar.xz
 
Set your root password
# ##i##passwd
>> Enter your password, you won't see what you are writing (for security reasons), but it is there!
 
Get out of the chroot environment
# ##i##exit
 
Unmount all the kernel filesystem stuff and boot (if you have a separate /boot)
# ##i##umount -l proc dev sys boot
 
Turn off the swap
# ##i##swapoff /dev/zvol/tank/swap
 
Export the zpool
# ##i##cd /
# ##i##zpool export tank
 
Reboot
# ##i##reboot
</console>
 
{{fancyimportant|'''Don't forget to set your root password as stated above before exiting chroot and rebooting. If you don't set the root password, you won't be able to log into your new system.'''}}
 
and that should be enough to get your system to boot on ZFS.
 
== After reboot ==


== Troubleshooting ==
=== Forgot to reset password? ===
=== Forgot to reset password? ===
==== System Rescue CD ====
You will need to chroot into your Funtoo on ZFS root installation if you forgot to set the root password. Repeat the earlier instructions to load the the live CD/USB, including the installation of ZFS kernel modules.
If you aren't using bliss-initramfs, then you can reboot back into your sysresccd and reset through there by mounting your drive, chrooting, and then typing passwd.
 
Example:
<console>
# ##i##zpool import -f -R /mnt/funtoo tank
# ##i##chroot /mnt/funtoo bash -l
# ##i##passwd
# ##i##exit
# ##i##zpool export -f tank
# ##i##reboot
</console>
 
==== Using bliss-initramfs ====
If you forgot to reset your password and are using '''bliss-initramfs''', you can add the '''su''' option to your bootloader parameters and the initramfs will throw you into the rootfs of your drive. In there you can run 'passwd' and then type 'exit'. Once you type 'exit', the initramfs will continue to boot your system as normal.


=== Create initial ZFS Snapshot ===
When the ZFS kernel modules are loaded, your existing ZFS Storage Pool (rpool) will be imported automatically. This will also result in an attempt to mount the various ZFS filesystems that you created. Mounting your root partition (/) will fail since this location is not empty; / is allocated to the live CD/USB distribution!
Continue to set up anything you need in terms of /etc configurations. Once you have everything the way you like it, take a snapshot of your system. You will be using this snapshot to revert back to this state if anything ever happens to your system down the road. The snapshots are cheap, and almost instant.
{{console|body=
###i## zpool status
  pool: rpool
state: ONLINE
  scan: none requested
config:


To take the snapshot of your system, type the following:
NAME                                            STATE    READ WRITE CKSUM
<console># ##i##zfs snapshot -r tank@install</console>
rpool                                            ONLINE      0    0    0
  ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF  ONLINE      0    0    0


To see if your snapshot was taken, type:
errors: No known data errors
<console># ##i##zfs list -t snapshot</console>
}}


If your machine ever fails and you need to get back to this state, just type (This will only revert your / dataset while keeping the rest of your data intact):
To get around this issue, first, manually export the pool.
<console># ##i##zfs rollback tank/funtoo/root@install</console>
{{console|body=
###i## zpool export rpool
###i## zpool list
no pools available
}}


{{fancyimportant|'''For a detailed overview, presentation of ZFS' capabilities, as well as usage examples, please refer to the [[ZFS_Fun|ZFS Fun]] page.'''}}
Now, import your storage pool using the following command.
{{console|body=
###i## zpool import -o cachefile=/tmp/zpool.cache -R /mnt/funtoo -d /dev/disk/by-id/ rpool
}}
This will import rpool and place your Funtoo install at {{f|/mnt/funtoo}}. Follow the earlier set of instructions to chroot into your Funtoo Installation on ZFS root.


== Troubleshooting ==
=== Will not mount on first reboot? ===
Follow the above instructions on "Forgot to reset password?" to import and mount your storage pool (rpool) and chroot back into your Funtoo environment.


=== Starting from scratch ===
Things to check:
If your installation has gotten screwed up for whatever reason and you need a fresh restart, you can do the following from sysresccd to start fresh:
* Was the ''bootfs'' property of rpool set? Use {{c|zpool get bootfs rpool}} to check
* Was /tmp/zpool.cache copied into /mnt/funtoo/etc/zfs/ prior to chroot and creating the initramfs?
* Was {{c|genkernel initramfs}} run with the --zfs switch and --kernel-config pointing to the correct configuration file?
* Was GRUB installed and configured correctly?


<console>
===  rpool is 'busy' on zpool export ===
Destroy the pool and any snapshots and datasets it has
This happens when swapon is used.
# ##i##zpool destroy -R -f tank
Swap must be turned off to free the pool for export.
 
This deletes the files from /dev/sda1 so that even after we zap, recreating the drive in the exact sector
position and size will not give us access to the old files in this partition.
# ##i##mkfs.ext2 /dev/sda1
# ##i##sgdisk -Z /dev/sda
</console>


Now start the guide again :).
=== Unable to add universe repository in Ubuntu? ===
See http://askubuntu.com/questions/761592/unable-to-apt-get-dist-upgrade-on-a-persistent-ubuntu-16-04-usb


[[Category:HOWTO]]
[[Category:HOWTO]]
[[Category:Filesystems]]
[[Category:Filesystems]]
[[Category:Featured]]
[[Category:Install]]


__NOTITLE__
__NOTITLE__

Latest revision as of 09:35, September 16, 2023

   Warning

This page is unofficial. ZFS as root filesystem is not supported under Funtoo Linux, mainly because it has limited benefit. ZFS is still supported but boot on a non-ZFS filesystem first. See ZFS.

Introduction

This wiki article will show you how to install Funtoo on ZFS (rootfs).

Prerequisites

   Important

ZFS is designed for 64-bit systems. We only recommend and support 64-bit platforms and installations!

   Warning

ZFS v07.5 (latest in ports as of 18 Jan 2018) is compatible with kernel versions 2.6.32 - 4.14

   Warning

The guide is under rewrite

It is recommended to give the entire disk to ZFS. As such, this guide will only show how to install ZFS on the whole disk, using legacy boot. Installing on UEFI requires a separate partition for /boot, formatted as FAT32, and is out of the scope of this guide, even though installation on UEFI is certainly possible. Also, this guide will not cover anything related to encryption (native encryption is available: https://github.com/zfsonlinux/zfs/commit/b52563034230b35f0562b6f40ad1a00f02bd9a05).

Downloading the ISO (With ZFS)

In order to install Funtoo on ZFS, you will need an environment, such as live media, that provides the ZFS tools. This guide will utilize the Ubuntu Desktop 16.04 (live) DVD for amd64.

  • Download from [1]

Creating a bootable USB from ISO (From a Linux Environment)

After you download the iso, you can do the following steps to create a bootable USB:

   Note

The size of the iso is approximately 1.5 GB.

Insert your blank USB media into a USB port. Then, inspect the kernel ring buffer with dmesg to identify the device name of your USB storage. 

[  +5.533491] usb 6-2: new SuperSpeed USB device number 4 using xhci_hcd
[  +0.022995] usb 6-2: New USB device found, idVendor=1b1c, idProduct=1a0c
[  +0.000006] usb 6-2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[  +0.000003] usb 6-2: Product: Voyager Mini 3.0
[  +0.000003] usb 6-2: Manufacturer: Corsair
[  +0.000002] usb 6-2: SerialNumber: 0123456789ABCDEF
[  +0.001095] usb-storage 6-2:1.0: USB Mass Storage device detected
[  +0.000080] scsi host15: usb-storage 6-2:1.0
[  +1.000772] scsi 15:0:0:0: Direct-Access     Corsair  Voyager Mini 3.0 PMAP PQ: 0 ANSI: 6
[  +0.000615] sd 15:0:0:0: Attached scsi generic sg6 type 0
[  +0.000110] sd 15:0:0:0: [sdg] 60566016 512-byte logical blocks: (31.0 GB/28.9 GiB)
[  +0.000209] sd 15:0:0:0: [sdg] Write Protect is off
[  +0.000004] sd 15:0:0:0: [sdg] Mode Sense: 2b 00 00 08
[  +0.000227] sd 15:0:0:0: [sdg] Write cache: disabled, read cache: enabled, doesn't support DPO or FUA
[  +0.359532] sd 15:0:0:0: [sdg] Attached SCSI removable disk

In this example, [sdg] indicates that the device is /dev/sdg.

A quick and easy way to create a bootable USB is to write the ISO data to the USB device using dd. 

root # dd if=/path/to/iso/ubuntu-16.04-desktop-amd64.iso of=/dev/sdg bs=4K

Once this has completed, remove and use this USB to boot the target system that will receive Funtoo Linux.

Booting the ISO

Using legacy (BIOS) boot mode, boot the ISO and allow Ubuntu to load the graphical environment. You will be presented with a "Welcome" dialog (titled Install (as superuser)). Select the option "Try Ubuntu".

Once the desktop has loaded, open the search bar by left-clicking on the top-left icon ("Search your computer"). Pressing Alt+F1 should also open this search bar. Search for and open the Terminal application.

In the terminal, issue the following commands to install and load the required ZFS module. 

ubuntu@ubuntu:~$ sudo -i
root@ubuntu:~# apt-add-repository universe
root@ubuntu:~# apt update
root@ubuntu:~# apt install --yes debootstrap gdisk zfs-initramfs

Verify that the ZFS kernel module has loaded. 

root@ubuntu:~#  dmesg | grep ZFS
[  377.595348] ZFS: Loaded module v0.6.5.6-0ubuntu10, ZFS pool version 5000, ZFS filesystem version 5

Drive Partitions

We are letting ZFS automatically partition the drive. This is ideal for our example using a single disk and legacy (BIOS) boot.

Creating of pool

Create a ZFS Storage Pool using a single whole disk. 

root@ubuntu:~# zpool create -d -o feature@async_destroy=enabled -o feature@empty_bpobj=enabled -o feature@lz4_compress=enabled -o feature@spacemap_histogram=enabled -o feature@enabled_txg=enabled -o feature@extensible_dataset=enabled -o feature@bookmarks=enabled -f -o ashift=12 -o cachefile=/tmp/zpool.cache -O normalization=formD -O atime=off -O xattr=sa -O compression=lz4 -m none -R /mnt/funtoo rpool /dev/disk/by-id/foo

The options used here are

optiondescription
createUse zpool to create a ZFS Storage Pool.
-dRequired. Disables enabling all availabe zfs features. This is needed as the GRUB bootloader currently can't handle all advanced zfs features, not specifying this flag on pool creation (or running 'zpool upgrade' on the pool at a later point in time) will make the pool incompatible with GRUB.
(multiple) -o feature@<name>=enabledExplicitely enable all features currently supported by GRUB. See 'man zpool-features' for details on their effects.
-fForce the use of the selected disk.
-o ashift=12Alignment of the pool to underlying hard drive sectors. The recommended value is 12, which corresponds to 2^12 Bytes or 4 KiB. This value is typical for present-day HDD's. Can only be set once at pool creation.
-o cachefile=/tmp/zpool.cacheCreate a pool configuration cache and place it in /tmp. This will be required for our Funtoo install.
-O normalization=formDRecommended. Set the default Unicode (UTF-8) normalization for future filesystems (created within this pool) to 'formD'.
-O atime=offRecommended. As a default preference, set future filesystems (created within this pool) to not update file access time. Useful if we want to reduce writes to disk (e.g., Solid State Drives). Can cause problems for mailers and other software that rely on file access-time data.
-O xattr=saRecommended. As a default preference, set future filesystems (created within this pool) to store extended file attributes in a more efficient manner.
-O compression=lz4Recommended. As a default preference, set future filesystems (created within this pool) to be compressed using the lz4 algorithm. Useful in general as the runtime cost for compression is neglible on these days while on-disk space still costs money.
-m noneDo not set mountpoint for this storage pool. (This guide will address this later).
-R /mnt/funtooAlternate root directory. Essentially a temporary 'mount point' for our pool.
rpoolThe name of this ZFS Storage Pool. The pool name is irrelevant. rpool will be used throughout this guide.
/dev/disk/by-id/fooThe path to the physical disk. e.g. /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF. Known in 'ZFS-speak' as a 'VDEV'.

Please note that the options 'ashift' and 'cachefile' are preceded with a lowercase '-o', while 'normalization' and 'atime' are preceded with an uppercase '-O'.

Without additional options, this will create our storage pool and enable all 'features' available under version 0.6.5.6. The pool will be automatically mounted at the (temporary) location /mnt/funtoo.

To confirm the presence of our newly created pool: 

root@ubuntu:~# zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME                                             STATE     READ WRITE CKSUM
	rpool                                            ONLINE       0     0     0
	  ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF  ONLINE       0     0     0

errors: No known data errors

Create ZFS Datasets

Now, we will create one or more ZFS datasets within our storage pool. These will contain Funtoo Linux.

Create the Root file system (Required)

root #  zfs create -o mountpoint=none -o canmount=off rpool/ROOT
root #  zfs create -o mountpoint=/ rpool/ROOT/funtoo

Create optional Datasets

The following optional datasets are provided as examples. It is up to the user to create their own datasets.

Home
root #  zfs create -o mountpoint=/home rpool/HOME
Build directory
root #  zfs create -o mountpoint=none -o canmount=off rpool/FUNTOO
root #  zfs create -o mountpoint=/var/tmp/portage -o compression=lz4 -o sync=disabled rpool/FUNTOO/build
Swap on ZFS

With some careful tuning a swap partition can be created on a ZFS 'volume' [2]. For a 2 GB swapfs: 

root #  zfs create -V 2G -b $(getconf PAGESIZE) -o logbias=throughput -o sync=always -o primarycache=metadata rpool/swap
root #  mkswap /dev/zvol/rpool/swap
root #  swapon /dev/zvol/rpool/swap

Note that swap on ZFS has known stability issues. If the user has decided to use swap, please take following into account: Always use long /dev/zvol aliases in configuration files. Never use a short /dev/zdX device name.

To confirm the presence of the filesystems that we have created: 

root@ubuntu:~# zfs list -t all
NAME                USED  AVAIL  REFER  MOUNTPOINT
rpool               660K  19.3G    96K  none
rpool/HOME           96K  19.3G    96K  /mnt/funtoo/home
rpool/ROOT          192K  19.3G    96K  none
rpool/ROOT/funtoo    96K  19.3G    96K  /mnt/funtoo

Make the root filesystem bootable

   Important

Do not skip this!

When booting from ZFS, you must specify a boot device and a root file system within the pool that was identified by the boot device. By default, the dataset selected for booting is the one identified by the pool's bootfs property. 

root # zpool set bootfs=rpool/ROOT/funtoo rpool

Installing Funtoo

Now that the initial ZFS pool and datasets have been created, we can unpack the stage 3 tarball and proceed with basic system installation. This part does not differ much from a regular Funtoo Linux installation (https://www.funtoo.org/Install#Installing_the_Stage_3_tarball). We recommend using funtoo-current stages for ZFS. 

root # cd /mnt/funtoo
root # wget https://build.funtoo.org/next/x86-64bit/generic_64/stage3-latest.tar.xz

Extract the contents with the following command. If the user is using an arch-optimized stage 3 tarball, substitute in the actual filename. 

root # cd /mnt/funtoo
root # tar xpf stage3-latest.tar.xz

Now, we need to create a chroot environment: 

root # cd /mnt/funtoo
root # mount -t proc none proc
root # mount --rbind /sys sys
root # mount --rbind /dev dev

An important step is to copy the ZFS cache into the chroot. The ZFS cache was created when rpool was created with the -o cachefile=/tmp/zpool.cache flag. 

root # mkdir -p /mnt/funtoo/etc/zfs
root # cp /tmp/zpool.cache /mnt/funtoo/etc/zfs/zpool.cache

You will also want to copy over resolv.conf in order to have proper resolution of Internet hostnames from inside the chroot: 

root # cp /etc/resolv.conf /mnt/funtoo/etc/

We are now ready to chroot. 

root # chroot /mnt/funtoo /bin/bash
root # export PS1="(chroot) $PS1"; cd

Configuring your system

Configure your system according to the main install guide. During startup, ZFS filesystems will be mounted without needing any entries in /etc/fstab. Comment out all entries in /etc/fstab except for partitions such as CD-ROMs, tmpfs, etc., if used.

If you created a swap volume earlier, add an appropriate entry to /etc/fstab. 

root #  echo /dev/zvol/rpool/swap none swap defaults 0 0 >> /etc/fstab

Note that swap on ZFS is unstable. If you decide to use swap, please take following into account: always use long /dev/zvol aliases in configuration files. Never use a short /dev/zdX device name.

Next, update the portage tree: 

root # ego sync
root # env-update
root # source /etc/profile

Installing ZFS userspace and bootloader

Installing the ZFS userspace tools and kernel modules

Install the ZFS packages and sync the portage tree, if required: 

root # ego sync
root # emerge --ask sys-fs/zfs

Once it has successfully merged, add the following services to the boot runlevel of OpenRC: 

root # rc-update add zfs-import boot
root # rc-update add zfs-mount boot

Add another two services to the default runlevel: 

root # rc-update add zfs-share default
root # rc-update add zfs-zed default

Create a ZFS-friendly initramfs

The Funtoo stage3 includes a linux kernel and initramfs. The initramfs is designed to mount and start Funtoo Linux on a variety of file systems. The initramfs contained within the stage3 will not mount and start Funtoo in our ZFS storage pool. We must create an updated 'ZFS-friendly' initramfs.

Optional: Update to the latest No results: 

root # emerge --oneshot sys-kernel/genkernel

Use genkernel to create an initramfs capable of mounting our ZFS Storage Pool via the --zfs switch. Adjust --makeopts according to the number of available threads: 

root # genkernel initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs
root ##g##* Funtoo Linux Genkernel; Version 3.4.40.11-funtoo##!g##
root ##g##*##!g## Running with options: initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs

root ##g##*##!g## Linux Kernel 4.5.2-1 for x86_64...
root ##g##*##!g## .. with config file /usr/src/linux-debian-sources-4.5.2/.config
root ##g##*##!g## busybox: >> Using cache
root ##g##*##!g## initramfs: >> Initializing...
root ##g##*##!g##         >> Appending base_layout cpio data...
root ##g##*##!g##         >> Appending auxilary cpio data...
root ##g##*##!g##         >> Copying keymaps
root ##g##*##!g##         >> Appending busybox cpio data...
root ##g##*##!g##         >> Appending modules cpio data...
root ##g##*##!g##         >> Appending zfs cpio data...
cp: cannot stat ‘/etc/zfs/zdev.conf’: No such file or directory
root ##y##*##!y## Could not copy file /etc/zfs/zdev.conf for ZFS
root ##g##*##!g##         >> Appending blkid cpio data...
root ##g##*##!g##         >> Appending modprobed cpio data...
root ##g##*##!g##         >> Compressing cpio data (.xz)...

root ##y##*##!y## WARNING... WARNING... WARNING...
root ##y##*##!y## Additional kernel cmdline arguments that *may* be required to boot properly...
root ##y##*##!y## add "dozfs" for ZFS volume management support
root ##y##*##!y## add either "real_root=ZFS" (bootfs autodetection) or "real_root=ZFS=<dataset>" to boot from a ZFS dataset

root ##g##*##!g## Do NOT report kernel bugs as genkernel bugs unless your bug
root ##g##*##!g## is about the default genkernel configuration...
root ##g##*##!g## 
root ##g##*##!g## Make sure you have the latest ~arch genkernel before reporting bugs.

Confirm the presence of the new initramfs: 

root # ls /boot/*genkernel*
/boot/initramfs-genkernel-x86_64-4.5.2-1

Grub expects the initramfs filename to be of the form: initramfs-${KNAME}-${ARCH}-${KV}. This guide assumes the usage of debian-sources. Tailor it to your specific kernel name, platform, and kernel version. Forgetting to rename initramfs-genkernel to initramfs-debian-sources will render the system unbootable. 

root # cd /boot
root # mv initramfs-genkernel-x86_64-4.5.2-1 initramfs-debian-sources-x86_64-4.5.2-1
root # ls /boot/*initramfs*
/boot/initramfs-debian-sources-x86_64-4.5.2-1

Installing GRUB 2

GRUB 2 must be built with support for ZFS Storage Pools on a single disk. This is achieved using the 'libzfs' USE flag. 

root # echo "sys-boot/grub libzfs" >> /etc/portage/package.use
   Note

If you have defined GRUB_PLATFORMS in your /etc/portage/make.conf, please ensure that it includes 'pc': e.g. GRUB_PLATFORMS="efi-64 pc".

This is required for booting in BIOS (non-UEFI) mode, as described in this guide.

Now, install GRUB: 

root # emerge grub

Configuring the Bootloader

When zpool created our storage pool (rpool), it created partitions under a GPT scheme. In order to boot Funtoo Linux on a GPT partion under legacy (BIOS) boot, sys-boot/grub requires a small partition, called a BIOS boot partition. By design, ZFS (zpool) left a very small unpartitioned space at the beginning of the disk. We will use sgdisk, which is part of sys-apps/gptfdisk, to format this free space into a BIOS boot partition. 

root # sgdisk -a1 -n2:48:2047 -t2:EF02 -c2:"BIOS boot partition" /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF

To avoid problems with GRUB, use partx to refresh the list of partitions that are 'seen' by the kernel. Do this for each drive (/dev/sda, /dev/sdb, etc.). 

root # partx -u /dev/sda

A quick check to verify that GRUB 2 sees/supports ZFS: 

root # touch /etc/mtab
root # grub-probe /
zfs

Installing GRUB2 to disk is as easy as: 

root # grub-install /dev/disk/by-id/ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF
Installing for i386-pc platform.
Installation finished. No error reported.

Now, it's time for us to create grub's configuration file. First, we must edit a few GRUB 2 settings in /etc/default/grub:

   Important

The following is required to boot Funtoo Linux on ZFS!

Replace the line

   /etc/default/grub - before
#GRUB_CMDLINE_LINUX=""

with

   /etc/default/grub - after
GRUB_CMDLINE_LINUX="dozfs real_root=ZFS=rpool/ROOT/funtoo"

Now, create GRUB 2 configuration file: 

root # grub-mkconfig -o /boot/grub/grub.cfg
Generating grub configuration file ...
Found linux image: /boot/kernel-debian-sources-x86_64-4.5.2-1
Found initrd image: /boot/initramfs-genkernel-x86_64-4.5.2-1
done

If grub-mkconfig cannot find the initrd image, manually add initrd to the generated /boot/grub/grub.cfg, below the linux line:

   /boot/grub/grub.cfg - adding initrd below the linux line
echo    'Loading Linux x86_64-4.5.2-1 ...'
linux   /ROOT/funtoo@/boot/kernel-debian-sources-x86_64-4.5.2-1 root=ZFS=rpool/ROOT/funtoo ro dozfs real_root=ZFS
echo    'Loading initial ramdisk ...'
initrd  /ROOT/funtoo@/boot/initramfs-debian-sources-x86_64-4.5.2-1

A quick check to verify that the GRUB ZFS module is in place: 

root # ls /boot/grub/*/zfs.mod
/boot/grub/i386-pc/zfs.mod

Final configuration

Configure your network according to the main installation guide.

Set the root password. 

(chroot) # passwd

Exit the chroot and export your ZFS storage pool. 

(chroot) # exit
root # umount -lR {dev,proc,sys}
root # cd /
root # zpool export rpool

Restart to boot into Funtoo Linux on a ZFS root!

After reboot

Snapshot

Take a snapshot of your Funtoo at installation. 

root # zfs snapshot rpool/ROOT/funtoo@install

The use of snapshots, including sending snapshots as a method of backup, are not covered in this guide. See Further Reading.

ZFS Adjustable Replacement Cache (ARC) size

The Adjustable Replacement Cache (ARC) is a fundamental part of ZFS. Refer to this article by Aaron Toponce for details on how to set up an ARC.

Without configuration, ZFS will use up to 50% of your memory (RAM) for the ARC. It is possible to change this maximum. There are different ways to achieve this on both a temporary and persistent basis. One such way is to create and edit the file /etc/modprobe.d/zfs.conf, which affects the ZFS kernel module.

   /etc/modprobe.d/zfs.conf - set maximum ARC size to 4 GiB
options zfs zfs_arc_max=4294967296

where zfs_arc_max is set to a value in Bytes. After configuring this file, re-generate the initramfs. Rebooting will then apply this change.

To apply this change immediately without a reboot, issue the command: 

root #  echo 4294967296 >> /sys/module/zfs/parameters/zfs_arc_max

Reference: https://wiki.gentoo.org/wiki/ZFS#ARC

After Kernel or ZFS updates

The default Funtoo Linux kernel (sys-kernel/debian-sources) does not automatically build a ZFS-capable initramfs. Similarly, the package sys-fs/zfs and its dependencies do not automatically build a ZFS-capable initramfs.

After each kernel update, you must recreate an initramfs. Similarly, sys-fs/zfs updates also require you to regenerate an initramfs. This is especially true for updates to sys-fs/zfs that introduce new storage pool features. Neglecting to update your initramfs will make your system unbootable.

First, rebuild sys-fs/zfs, sys-fs/zfs-kmod and sys-kernel/spl: 

root # emerge --ask --verbose -1 zfs zfs-kmod spl

Then, follow the earlier instructions to create a new initramfs: 

root # genkernel initramfs --no-clean --no-mountboot --makeopts=-j4 --kernel-config=/usr/src/linux/.config --zfs
root # cd /boot
root # mv initramfs-genkernel-x86_64-4.5.2-1 initramfs-debian-sources-x86_64-4.5.2-1

Finally, repeat the earlier instructions for generating /boot/grub/grub.cfg and manually adding the new initramfs as initrd to the generated /boot/grub/grub.cfg.

Further Reading

ZFS has many interesting features not covered by this guide.

Useful information and instructions can be found in the online reference manuals. See man zpool and man zfs.

Aaron Toponce's Zpool/ZFS Administration Guides - https://pthree.org/2012/12/04/zfs-administration-part-i-vdevs

ZFS on Linux - http://www.zfsonlinux.org

ZFS - Gentoo Wiki - https://wiki.gentoo.org/wiki/ZFS

Troubleshooting

Forgot to reset password?

You will need to chroot into your Funtoo on ZFS root installation if you forgot to set the root password. Repeat the earlier instructions to load the the live CD/USB, including the installation of ZFS kernel modules.

When the ZFS kernel modules are loaded, your existing ZFS Storage Pool (rpool) will be imported automatically. This will also result in an attempt to mount the various ZFS filesystems that you created. Mounting your root partition (/) will fail since this location is not empty; / is allocated to the live CD/USB distribution! 

root # zpool status
  pool: rpool
 state: ONLINE
  scan: none requested
config:

	NAME                                             STATE     READ WRITE CKSUM
	rpool                                            ONLINE       0     0     0
	  ata-Samsung_SSD_840_EVO_120GB_123456789ABCDEF  ONLINE       0     0     0

errors: No known data errors

To get around this issue, first, manually export the pool. 

root # zpool export rpool
root # zpool list
no pools available

Now, import your storage pool using the following command. 

root # zpool import -o cachefile=/tmp/zpool.cache -R /mnt/funtoo -d /dev/disk/by-id/ rpool

This will import rpool and place your Funtoo install at /mnt/funtoo. Follow the earlier set of instructions to chroot into your Funtoo Installation on ZFS root.

Will not mount on first reboot?

Follow the above instructions on "Forgot to reset password?" to import and mount your storage pool (rpool) and chroot back into your Funtoo environment.

Things to check:

  • Was the bootfs property of rpool set? Use zpool get bootfs rpool to check
  • Was /tmp/zpool.cache copied into /mnt/funtoo/etc/zfs/ prior to chroot and creating the initramfs?
  • Was genkernel initramfs run with the --zfs switch and --kernel-config pointing to the correct configuration file?
  • Was GRUB installed and configured correctly?

rpool is 'busy' on zpool export

This happens when swapon is used. Swap must be turned off to free the pool for export.

Unable to add universe repository in Ubuntu?

See http://askubuntu.com/questions/761592/unable-to-apt-get-dist-upgrade-on-a-persistent-ubuntu-16-04-usb


Retrieved from "https://www.funtoo.org/index.php?title=ZFS_as_Root_Filesystem&oldid=43304"