System resurrection

Although it is always possible to resurrect a machine back to life by reinstalling it, it is not always suitable to reinstall from scratch. Sometimes the best approach is to try to repair, or "resurrect," a broken system so that is it fully functional again. This document will show you how to resurrect a broken Funtoo system without reinstalling everything from scratch.

Building binary packages

The best approach for critical system repair is to boot on a SystemRescueCD or other similar LiveCD, and use this as working platform to resurrect your system.

For creating binary packages, you can use any of the following for a source environment:

Use the most recent Funtoo stage3 for your architecture
If available, use a recent system backup (tar or cpio archive) or snapshot

On a working Linux system, which can be either your broken system booted with a LiveCD, or any type of Linux system on your network with similar processor, you will want to build a chroot environment using the "source environment" you selected above, and use this as a platform for building binary packages to restore your system. Once these packages are created, they can be copied to your broken system and installed using the steps later in this document.

The binary package creation environment would typically be set up as follows:

root # install -d /mnt/rescue
root # tar xpvf backup.tar.bz2 -C /mnt/rescue
root # cp /etc/resolv.conf /mnt/rescue/etc
root # mount --bind /proc /mnt/rescue/proc
root # mount --bind /sys /mnt/rescue/sys
root # mount --bind /dev /mnt/rescue/dev
root # mount --bind /dev/pts /mnt/rescue/dev/pts
root # chroot /mnt/rescue
root # source /etc/profile
root # env-update

No matter of the way you jump in a functional Funtoo/Gentoo environment, the magic command to create a binary package archive once inside is to use the quickpkg command. quickpkg will capture the package in the exact form it is actually deployed on the environment and create an archive of it placed in /usr/portage/packages/<package-category>/<package-name>-<package-version>.tbz2.

In the following example capture everything installed within the "source environment" that is related sys-devel/gcc (4.4.5 is present on the system) is captured in a single archive named gcc-4.4.5.tbz2 located in /usr/portage/packages/sys-devel:

root # quickpkg sys-devel/gcc

If you need to recompile a package instead of archiving an already deployed version (and of course without installing it on your "source environment"), just do:

root # emerge --buildpkgonly sys-devel/gcc

Restoring the binary packages on the broken system

There are a couple of methods that can be used to restore binary packages to a broken system.

Chroot/Emerge Method

This first approach can be used for lightly damaged systems that still have a functional Portage and to which you can still chroot and perform all basic Linux commands. To use this method, you would mount your broken system to /mnt/broken using steps similar to the way we set up /mnt/rescue, above.

Before or after chrooting, copy the binary packages created in the step above in the exact same location on your broken system (e.g. in /usr/portage/packages/sys-devel in the case of sys-devel/gcc).

Once chrooted inside your system, you will be able to merge your packages using emerge as follows:

root # emerge -k  sys-devel/gcc

Tbz2 Extract Method

This alternate method for installing binary packages is simpler and does not require the use of the chroot command. To prepare for using this approach, you just need to mount all key filesystems to /mnt/broken as follows -- and bind mounts are not necessary:

root # install -d /mnt/broken
root # mount /dev/sdaX /mnt/broken
root # mount /dev/sdaY /mnt/broken/usr

Then, use the following commands to extract the .tbz2 archive to your broken filesystem that you mounted at /mnt/broken:

root # tar xjpvf gcc-4.4.5.tbz2 -C /mnt/broken

You will see a note about the trailing garbage at the end of the file being ignored. This is normal -- tar is ignoring the Portage .tbz2 metadata that is tacked on to the end of the file.

At this point, you can set up bind mounts (see /mnt/rescue example earlier in this document for those steps), chroot inside /mnt/broken, and perform a few tests to determine if your issue has been resolved.

Fixing broken portage

Broken portage only

Sometimes sys-apps/portage may fail and the above binpkg mentioned way to resurrect system will not work because of portage being broken itself. To fix portage manually the following steps required:

root # cd /tmp
root # wget http://distfiles.gentoo.org/distfiles/portage-2.2.26.tar.bz2
root # tar -xjf portage-2.2.26.tar.bz2
root # rm -fr /usr/lib/portage /usr/lib*/python*/site-packages/{_emerge,portage,repoman}
root # /tmp/portage-2.2.26/bin/emerge -atv1 sys-apps/portage

Broken python and portage

A more extreme case, when both python and portage are broken. It's hard to determine if they really both broken, so this method suitable for updating a very old boxes, with dated python and portage versions. <console>

##i##cd /tmp
##i##wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tar.xz
##i##tar -xJf Python-3.5.0.tar.xz
##i##cd Python-3.5.0
##i##./configure --enable-shared --with-system-expat --with-system-ffi
##i##make
##i##cd /tmp
##i##wget http://distfiles.gentoo.org/distfiles/portage-2.2.26.tar.bz2
##i##tar xvf portage-2.2.26.tar.bz2