Netboot setup with hybrid (ro/rw) root filesystem

This page describes a netboot setup that is most appropriate in cases where several or many computers or VMs need to boot very similar or identical Gentoo setups. Typically this is the case for clusters or virtual server installations

Goals
A number of (diskless) computers or VMs will boot from a single server that supplies storage space. Each server mounts a common read only filesystem. Several parts of this filesystem are made rw-accessible by means of an AUFS2 overlay on the server in order to allow per-client-modifications. the most important point of this setup is that updates of the base system can be made server side in a chroot environment. Those updates are immediately effective for the netboot nodes.

Used Software

 * dnsmasq (for DHCP, DNS, and TFTP)
 * NFS Server
 * AUFS2
 * PXELinux

Setup instructions
This guide starts expecting the server to run a working Gentoo installation. In /var enough free space for the common client installation as well as the client specific overlays must be available. In this guide a setup of one server and four clients will be shown. The five computers are connected by means of an internal network. The server has an additional connection to the internet and serves as a router for the netbooting clients. The internal network uses the subnet 10.254.254.0/24. The server (pac-control) uses the static address 10.254.254.1 while our four clients get the addresses 10.254.254.10-13 via dhcp and are called pac1-4.

Folder creation
First we need a place to drop the netboot core system that is used by all clients as well as the client specific overlays. Additionally we need a folder to combine these two parts for later NFS exporting:

The and  folders need per-client subfolders that get names according to the IP addresses (e. g. ). Additionally we create a folder that holds the files accessible via TFTP:

Client system installation
A stage 3 snapshot of your choice needs to be extracted to the folder as described in the Gentoo Handbook. e.g.

To save hard disk space we bind-mount the portage tree into this folder:

To enable the installation of the base system there also needs to be a /proc filesystem in the netboot path:

To make it unnecessary to copy the kernel around we bind-mount the netboot-tftp directory:

And to get rid of another rw-folder we move /root into /home:

To get around boot problems concerning mtab (see below) a link is added:

After this step the client installation can be continued as described in the Gentoo Handbook. Begin with:

A kernel must be installed and built using genkernel. The following invocation produces a working kernel image and initrd:

Make sure to enable NFS root fs support:

dnsmasq
Dnsmasq supplies the necessary DHCP/DNS/TFTP infrastructure for automatic configuration of the clients. Here the basic configuration for our setup without comments is shown:

It is favorable to use to enable proper name resolution for internal and external hostnames alike:

PXELinux
We use PXELinux as "boot manager" for the netboot clients (as grub is not a good choice in this case!).

Some PXELinux files must be available via TFTP. Therefore we create symlinks:

Then PXELinux needs configuration files. We use a per client configuration as every client needs to mount its own NFS export:

Such a file must be created for every client computer. The subdirectory of must be changed according to the client's IP address.

The part enables PXELinux output on the second serial port which can be useful if the clients support IPMI or similar methods to access one of the serial ports via LAN.

AUFS2
AUFS is used to build the overlays for rw directories. it can be installed in the following way:

After this step the server's kernel needs to be rebuilt (emerge exits with an error message). Afterwards the emerge must be run again. This time it will work.

fstab (Server)
In order to mount everything to the right place the following fstab can be used on the server:

Make sure to create all directories before mounting!

NFS
Make sure that NFS server support is activated in the server's kernel. Install :

We have to export the trees below. This can be done using this settings:

The writeable subdirectories unfortunately have to be exported separately because they are mounted as separate "filesystems". The option makes it possible that the writeable subdirectories are accessible (at least in read only mode) as soon as the root fs is mounted.

Then the NFS server must be startet:

fstab (Client)
This fstab for the client computer works with the hacked scripts below.

Hacks to make it work
NOTE: According to this forum post  and this bug  this is a problem in the initramfs, which genkernel creates (specifically in the file /usr/share/genkernel/generic/linuxrc ). genkernel seems to be patchable against this so that this hack is not necessary.

Due to nasty problems concerning on non writeable filesystems (/etc is not writeable during the boot process) some boot scripts need to be changed. We used baselayout-2 in the client installation (in order to avoid that this bug makes it even worse) and therefore the shown scripts are changed baselayout-2 scripts. First install baselayout-2 according to this guide:

Then replace the checkroot script by this version (which is much shorter):

And replace the part of the localmount script with this version:

If you added additional rw parts to the filesystem you also have to add them to the localmount script!

Additional notes
If you successfully followed this guide your client computers should now be able to boot via PXE (of course it must be enabled in the bios/network card). Please feel free to drop a note if anything important in this guide is still missing. I am successfully running the described setup but it is possible that I simply forgot some steps. Mnagl 21:19, 1 May 2010 (GMT)

similar HOWTOs
here is a list of hopefully helpful tutorials also covering nfs-boot
 * http://www.gentoo.org/doc/en/altinstall.xml#doc_chap4
 * http://www.gentoo.org/doc/en/diskless-howto.xml