Build Your Own LiveCD or LiveDVD

Introduction
This article will show you how to create your own LiveCD. This method has the following advantages over Gentoo's own Catalyst tool:


 * The build source will not be deleted between iterations of LiveCD creation. This will allow you to periodically sync, update, merge and customize your environment the same way you do it with a real system.
 * The CD will boot using GRUB, not ISOLINUX, though ISOLINUX is still possible. This gives you the same flexibility you have with a real system, such as changing kernel parameters, discovering devices, et cetera.
 * Keeping your build source between iterations reduces the subsequent build times enormously.
 * If made correctly, the CD can boot on 32MB RAM
 * Things that are in a green box are scripts that will be used more than once, like build scripts. Those are likely to get updated once in a while too.

Requirements
You need sufficient disk space on your system. This will vary depending on which packages you include in your LiveCD (more packages increases the size). Typically 4GB free is sufficient (note that during devellopment/building it can be multitudes of 4GB).

You must be familiar with installing Gentoo using a stage3 tarball. If in doubt, please consult the Gentoo Handbook.

A healthy understanding of the workings of Gentoo and Linux in general is recommended. Things can get especially nasty with the new build script if you get it wrong.

You'll also need app-cdr/cdrtools and sys-fs/squashfs-tools installed to build properly.

Setting up the build environment
The build environment is a directory that will be "chrooted" into when the LiveCD is being created, much like a normal installation is from a LiveCD. The build environment will be inside another directory containing everything.

For convenience, the main directory can be assigned to a shell variable, such as LIVECD, with the build environment as the "source" directory within it.

Download the latest stage3 tarball from one of the Gentoo mirrors at http://www.gentoo.org/main/en/mirrors.xml to the livecd directory and untar it in the source directory.

Next, download the latest portage snapshot to the livecd directory and untar it in.

If you need to download packages during the installation, setup the necessary files in that are required to connect to the internet, such as.

Installing Gentoo
Set up necessary mount points before chrooting to your build directory.

To avoid re-downloading distfiles for the installation, bind the local distfiles directory to our chroot's. In this case, it is assumed that distfiles are kept in the default directory /usr/portage/distfiles.

Now chroot to source and start the installation. You might want to add 'linux32' before chroot if working from an 64bit to a 32bit install q)

One of the first things to do is modify /etc/make.conf. A few notes on USE variables:


 * The more you use, the bigger your environment will get. It might even become too big to fit on a CD!
 * Consider some negatives to save space if you want to avoid pulling in extra dependencies. Flags such as -X and -doc are probably essential (unless you really want to include a GUI or man pages).
 * Make sure you add livecd to your USE variables. This, along with a 'cdroot' parameter that we'll pass to the kernel at boot time, will make init scripts aware that we're booting from a CD so they don't do inappropriate things like checking the root filesystem.
 * Consider adding the minimal and userlocales USE flags (don't forget to edit /etc/locales.build).
 * Consider to set some Portage features in make.conf like: FEATURES="nodoc noinfo noman"

Here are the installation steps roughly:

Before merging the kernel and boot loader, important files in /etc must be fixed first. First, /etc/fstab:

Notice that the root is being mounted over a loop device with filesystem type squashfs. More on that later. Now, modify other /etc files as you see fit (e.g. hostname, rc.conf, conf.d/* and so on). It is assumed that, since you have set up a Gentoo system before, you know how to do this.

It is for example a good idea to change the keyboard layout one time : edit /etc/conf.d/keymaps. For french keyboard including € character support, change to the following layout :

If you want network support, edit /etc/conf.d/net. This example sets dhcp configuration for eth0:

Notice that dhcpcd must be merged.

Kernel and initrd
Then it's time to install the kernel sources and configure them.

If you use the vanilla sources you should use >=2.6.29 kernel sources, otherwise you have to patch them with the squashfs-patch from http://squashfs.sourceforge.net.

Note however that the above target enables CONFIG_CMDLINE_BOOL in conjunction with CONFIG_CMDLINE_OVERRIDE which means that bootloader's kernel parameters will be ignored in favor of those specified in CONFIG_CMDLINE, which is empty by default. In general, it would be a good idea to review the generated kernel .config for conflicts and issues like this.

Now merge your desired kernel. We are using gentoo-sources(2.6). As we are going to use squashfs as our root filesystem, be sure to patch your kernel with it if you are not using a standard gentoo-sources / vanilla >=2.6.29.

When configuring your kernel (see below, using genkernel --menuconfig), make sure the following is compiled in:

If your planning boot through usb, such as on a flash drive, you´ll need to compile the kernel with the USB Mass Storage support:

IEEE 1394 ethernet support may disturb your system : sometimes an ethernet interface is recognized first, that is to say before "Classic" ethernet interface. If you don't need IEEE ethernet support, don't activate it :

Note: If you want to let make mkxf86config.sh an usable xorg.conf at startup, you have to build usbmouse ps2mouse etc as modules (and not compiled in!) since the script is searching for them.

Optionally, you can add any convenient option you use. For example, Magic SysRq Keys, allows rebooting instantly when no file system is mounted (or you might get corrupted filesystems). Linux Kernel Configuration: kernel configuration3

We wanted something like the gentoo 2005.0 as it is much more flexible, looks 1000x better, and boots more systems. After some research, found that it was genkernel that makes such an initrd possible. (believe me, using the initrd from the 2005.0 cd is not possible today).

The old can still be used, but is not recommended. Also, this howto is now based on genkernel.

If you have already configured AND compiled the kernel, tell genkernel to only create initrd by running:

Otherwise if you have not compiled kernel, do the following:

Or leave the menuconfig bit if you have a working .config.

Then do not forget to copy/rename the file to reflect the GRUB configuration (remember that as stated before, iso9660 doesn't handle symbolic links). For example, if you are working with a 2.6.37-gentoo-r4 type :

We now have a suitable kernel and ramdisk.

Cleaning some files
It is a good idea to clean the kernel tree, keeping only ".config" file:

Remove all the locales you don't need. The way to do this is to emerge the localepurge package:

And specify which locales you wish to keep in /etc/locale.nopurge:

Also, make sure to remove the NEEDSCONFIGFIRST line from the top.

Then run localepurge and re-index:

Delete wrong symbolic links :

Finally, delete ".keep" files that are necessary to restore empty directories from stage2 and Portage TAR archives:

Using "locate"
Locate is a convenient and fast tool to know if some file is present on your system. If you want to use it, you have to build the file database:

cat /proc/mounts is needed for the updatedb command. If not done, you get: updatedb output

Then you'll be able to search a file using locate:

locate output

Installing and configuring the CD bootloader
The final step is merging and configuring the boot loader. We are going to use GRUB since it has support for booting a cdrom.

Also, delete the symlink /boot/grub/menu.lst and then move grub.conf to menu.lst. This is because iso9660 can not handle symlinks. If you don't, grub will just give a dumb commandline!

Final cleanings
Inside the chrooted environment, don't forget to remove old kernel modules. For example:

Command output

Here we can remove 4 obsolete modules directories (delete all directories except the kernel(s) the LiveCD shall be booting):

We can also remove old source trees:

Then we can leave the chrooted environment:

Make a copy of our root source tree. This way we will be able to make more aggressive deletions on the copy without harming the original which can be useful for updates.

If you plan to use SYSLINUX, also do this:

Copy the source tree. It will not make its way onto the image, so don't worry about the size.

The following files can be skipped. Either by adding --exclude arguments to rsync, or running rm commands afterwards:


 * var/run/* (but not the directory itself)
 * var/tmp/* (but not the directory itself)
 * var/cache/* (but not the directory itself)
 * var/log/* (but not the directory itself)
 * usr/portage
 * etc/portage
 * usr/share/doc
 * usr/src
 * root/* (but not the directory itself)
 * tmp/* (but not the directory itself)

More aggressive cleanings
The goal here is to remove all software that is not mandatory. For example portage, automake and gcc will be useless on the LiveCD.

Inside the chroot do:

This will provide a complete list of the desired packages.

Outside the chroot and after building the target execute a script like this:

This will remove files, symbolic links, and empty directories (as stated, rmdir will only remove empty directories)

devsk: Are the static libraries needed on the LiveCD? I saved about 85MB by doing

find ${LIVECD}/target/files/source/usr/lib -name "*.a" -print -exec rm {} \;

Better yet, you can put "*.a" in the rsync --exclude command line.

Building the LiveCD
First, using squashfs tools, convert the source directory to a squashed image. Install the sys-fs/squashfs-tools package first if you don't have it.

Next, create the required empty livecd file. This file must be on livecd root, because the init script in initramfs uses this file to identify if the CD is mounted or not.

Finally, make the ISO image.

Now you can:


 * 1) Test livecd.iso using qemu; see section Testing the LiveCD without wasting a blank CD
 * 2) Burn livecd.iso

FINISHED :D

Updating the CD
You can chroot again to the CD tree and make some updates:

Now you can go back to the step "Cleaning some files" and follow the step to create a new ISO image.

Tips
Mounting /dev is important to get things like X to work from a chroot environment.

Another thing to consider is the size of your build source. As you can see, there're things that take a lot of space that are strictly not needed on a LiveCD. Two major candidates are /usr/src and /usr/portage. If you don't want to include them on your LiveCD don't delete them, you can utilize an option in mksquashfs to exclude any directory from the squashed image. This will keep your build source directory intact.

If booting the CD fails at some point during linuxrc script execution, you can always change your grub line to init=/bin/sh instead of linuxrc. This will give you a nice little shell inside the initrd image at boot time. From there you can type the commands in the linuxrc script manually and try to find out with one is failing and why.

I hope I didn't miss anything, and hope you find this howto useful. Your corrections/feedback are always welcome.

It is not necessary to bootstrap the system; the cd wil be able to boot from non-bootstrapped or bootsrapped systems.

Generate Xorg.conf at startup
Note: mkxf86config realy depends on hwsetup, so you need to edit /etc/init.d/mkxf86config and add /usr/sbin/hwsetup before /usr/sbin/mkxf86config.sh. It is also recomended to use newest versions of mkxf86config and hwsetup (currently lastest is mkxf86config 0.9.8 and hwsetup 1.2, but you need to write ebuild manualy). If you want mkxf86config to detect hardware properly - you need to edit /etc/init.d/mkxf86config or it will not detect your hardware. Also it is useful to look through /usr/sbin/mkxf86config.sh... If you are building LiveCD for x86 compatible PCs - you can uncomment ddcxinfo-knoppix and emerge this tool (ddcxinfo-knoppix - it will automaticlly probe monitor for information, but it may have some troubles with old and non-x86 hardware

Comment-alchark: hwsetup will actually be called by autoconfig script from the livecd-tools package, and it will also call mkxf86config as required.

emerge -av livecd-tools rc-update add autoconfig default

--end comment

Auto Mounting partitions spiced up
I have made an automounter script that looks up which partitions are available and then tries to mount them. But it is not a stupid loop.

1. find all available partitions 2. skip unmountables such as extended and grep error's 3. use swap as swap instead of mounting 4. mount ntfs with captive-ntfs 5. if none of the above, just try to mount

Update: Use ntfs-3g to avoid licensing issues and have much faster/better ntfs support.

I do this by greping fdisk -l, and using a second loop in my loop (because of the * that fdisk puts in front of the bootable partitions).

I have put my script online here, do not edit it using notepad or wordpad because they put a ^M at the end of each line which will break your LiveCD completely!

I could use some help perfecting the 2>/dev/null. It still displays some output sometimes.

* Try &>/dev/null, this way both STDOUT and STDERR are sent to nirvana. LX

* Maybe this script will help to detect partition types more accurate, it's taken from Knoppix and changed, there is another but a little worse tool (may miss fstype for example on promise fasttrack and other raid controllers): guessfstype from SuSE [mf]

Auto Login
If you want root to be automatically logged in, there's nothing to do, because usually the inittab is fixed during the boot in order to autologin root on a LiveCD.

However, many LiveCDs are intended to show Linux novices the principles of a Linux system, so it might be undesired to autologin as root, which is a large security threat. So if you want to autologin as non-privileged user, here's how to do it.

First emerge mingetty. Then edit /etc/inittab on the cd (be sure to make a backup), find:

replace with

for every tty you want autologon for a user named "joe".

If you want to display an extra message next to "OpenRC 0.2.2 is starting up Gentoo-Linux(i686)" now is the time to edit openrc-0.2.2/src/rc/rc.c; Extract ex /usr/portage/distfiles/openrc-0.2.2.tar.bz2 to a test location, then edit;

After this we need to get this back to the tar.bz2 and get it verified

Fix /root to ~
Edit /root/.bashrc env-update source /etc/profile cd ~

No Auto Login
To enable login for something like Mini Terminal you can also edit /sbin/livecd-functions.sh in your /home/root/livecd environment:

echo "c${x}:12345:respawn:/sbin/agetty -nl /bin/bashlogin 38400 tty${x} linux" >> /etc/inittab echo "c${x}:12345:respawn:/sbin/agetty 38400 tty${x} linux" >> /etc/inittab
 * 1) change to

echo "s0:12345:respawn:/sbin/agetty -nl /bin/bashlogin ${LIVECD_CONSOLE_BAUD} ${LIVECD_CONSOLE} vt100" >> /etc/inittab echo "s0:12345:respawn:/sbin/agetty ${LIVECD_CONSOLE_BAUD} ${LIVECD_CONSOLE} vt100" >> /etc/inittab
 * 1) change to

Optimizing the booting process
Note: No optimization is required if app-misc/livecd-tools is installed (in the chroot environment). That package provides scripts to detect whether a LiveCD is running and to disable unneeded instructions like the ones described below.

I have removed these optimizations since they were obsolete for some time and the new baselayout2 and openrc made them completly obsolete.

Another neat trick: edit /etc/rc:

Change rc_parallel="no" to rc_parallel="yes"

This will start all the services at the same moment, not one after another. Some people say this has no effect yet, but it looks cool and maybe there will be a time it has a purpose. But when using autoconfig from the LiveCD tools it will break your boot sequence.

"Defragmenting" the LiveCD to make it faster
By default, the files inside the squashfs filesystem are stored continuously, but unordered: this means that the cdrom drive will potentially have to seek them at very distant disk offsets, leading to higher cold start read times -- this is especially apparent at boot time if the LiveCD contains desktop environments such as GNOME or KDE, or applications such as Firefox or GIMP that use large number of files.

Fortunately, mksquashfs comes with a -sort option that makes it possible to change files' order on the filesystem. The easiest way to build the needed sort file is to include openlog in your LiveCD kernel image, then run the resulting LiveCD. openlog makes the kernel able to timely log all the successfully opened files; it also includes a script, openlog.py, that makes it easy to reuse this information to create the sort file. openlog comes fully documented: see the README or README.html file provided with it.

Note: I have moved the openlog page to my own host since I have been unable to contact the author, also I have added a new patch file for newer kernel versions.

The quick and dirty howto:

Boot the LiveCD with logfiles=1 and get dmesg from it to your build environment and extract openlog.py as well. File: dmesg

And change your build script to let the ordering take effect in the next build

Checking/editing the Initramfs
If you want to edit or check the initramfs generated, you can do something like this.

or

-- Thanx to robbat2

Initrd:

USB Stick drive
(evildick 2005-08-09)


 * IMPORTANT NOTE: usbroot is NOT a valid kernel option, and removing the cdroot option will cause boot failure (/dev/loop0 will fail to mount on /newroot. I also prefer to use parted & fat32 instead of sfdisk and ext2 - Conti


 * For the most part this works. Problems may be experienced due to bios limitations. Problems can range but usually it will just lock up shortly around initialization of the kernel (during or on the initramfs). This so far only occurs with a Supermicro P4SCi (it is likely that it will also occur with a P4SC8 - similar boards, but I have not tested it)
 * The script was designed to be used in conjuction with the above LiveCD scripts (this script being used last or replacing the mkiso command). The script above will format your usb device with an ext2 filesystem and write the squash image along with the grub loader files to it. After copying the grub boot loader menu it will alter it to work with the usb stick as a bootable device. Be sure to change the device settings in beginning of the script to reflect the mount directory you wish to use, as well as the usb stick device. You may also need to compile your kernel with USB devices (UHCI and OHCI) not as modules.

PXE Booting
Using the above instructions to build a squash image you can boot into the image via PXE (grub works but does not support as many network cards as pxelinux). NFS and tftp are required on the server end for this method. The following is required to be enabled in the kernel on the LiveCD or client end.

Note: : Don't forget to add to your kernel built-in (no modules!)


 * NFS filesystem client support as ;
 * NFS ROOT support ;
 * your network card(s) support ;

Copy the contents of your target directory into your tftp/nfs directory. Use the following parameters in your PXELinux default configuration file:

Other methods to Copy files

 * I have heard that tar and cpio are supposed to copy files faster than cp. I have not confirmed this yet, but here is a little bit of code to try out if you like.
 * Comment out the line in your build script:

And add this above or below it:

(Comment - evermind) why don't you use tar cpf - * before the pipe? Isn't that neccesary to preserve the permissions? (End Comment)

(Comment - letum) The p option is needless. Yes, it preserves permissions, but only when you extract the archive. Tar saves permissions by default - p is necessary only during the extraction process. (End Comment)

I have not fully tested this, so be careful. You can also add; '--exclude=usr/src --exclude=var/tmp* --exclude=usr/portage exclude=etc/portage --exclude=var/db --exclude=tmp/* --exclude=var/cache/*' to the first tar if you don't mind leaving the unemerge programs in that the before-build.sh scripts removes. This cuts the time down quite a bit.

Tested it, it seemed to work, but tar does not retain the permissions correctly so in the end it is a bad choice. Which is too bad, because it was fast. I am currently looking into cpio. (man tar : tar [ -p, --same-permissions, --preserve-permissions ]

Also, rsync works very well as it only updates what has been updated.

Note: the following tube should work all the time. With the first line, it makes a mini-script that is as error-safe as can be. It should preserve ownerships.

Testing the LiveCD without wasting a blank CD
If you're trying out some weird stuff with your LiveCD and don't want to waste a blank CD (and don't have a rewritable) each time you've built a new ISO-image, you can emulate the LiveCD with qemu, a very nice open-source emulator.

If you don't have qemu already, of course you need to emerge it (kqemu is an accelerator module)

1. Notice from falcon15: Perhaps you should do 'echo "app-emulation/qemu kqemu" >>/etc/portage/package.use' before merging qemu in order to make use of kqemu

To use it, there are several possibilities:

modprobe kqemu
 * 1) first, load the kqemu kernel module
 * 2) in order to get /dev/kqemu in udev on most distros you'll need to do:
 * 3) modprobe kqemu major=0

qemu -cdrom livecd.iso
 * 1) simple example of qemu usage

nice -n 2 qemu -cdrom livecd.iso
 * 1) give more resources on the host systems (you need to be root for this)

qemu -cdrom livecd.iso -m 512
 * 1) give more (emulated) memory, here: 512 megs

qemu-system-x86_64 -m 512 -usb /dev/sde
 * 1) booting from usb flash drive identified as /dev/sde on x86_64
 * 2) note: do an ls -l on the device in dev first, as it may be necessary to add your user to a group with read/write privilege to the device

It may be that some exotic functions are not supported by qemu, yet this shouldn't happen too often. If emulation starts failing at a very basic point, you might want to try emulating something like a Knoppix image in order to see if your qemu install is ok. However, booting sometimes hangs for about ten minutes or even more at

>>> Copying read-write image contents to tmpfs

This is due to qemu's memory management. But apart from the waiting, this is a harmless effect, and it doesn't seem to occur on a burned LiveCD.

Making a bootable floppy boot from a boot option in your CD's bootloader
This is very simple after I figured this out, you just need a few basic things:

Get syslinux:

Copy the memdisk to your boot folder:

Make an image of your bootable floppy:

And after all these hard steps just make a boot option (example in grub)

and add

That's all there is to it!

32bit chroot on a 64bit installation
Important note: if you are following this guide from within a 64-bit environment, consult the "32Bit Chroot Guide for Gentoo/AMD64" at [3] for how to properly chroot.

To modify the build script change (the fast way);

chroot ${TARGET_SOURCE} /bin/bash --login <<CHROOTED to linux32 chroot ${TARGET_SOURCE} /bin/bash --login <<CHROOTED

Comment: If you don't have linux32 at hand, you could also just modify the following: make.conf for right architecture, run make ARCH=i386 (menuconfig) or genkernel --arch-override=x86 initrd/all, (depends on what you use)!

Floppy tips
If you have a floppy that you have made bootable, but you need to make it bigger, so more files will fit on it, that can be done very simply.

Create a new file as big as you need it to be.

Make it a file system (dos for this instance).

Mount the new image and fill it (redo the above steps if you still have not got enough space).

Fill it with the files you need. The next step is to get the old boot sector (original_floopy can be a dd image or a /dev node).

Then simply merge the bootsector and new_image;

After that you can simply start it as the old floppy!

This can also be done for hd's. I recommend using something like VMware. Make a small hd and install whatever you want on it (needs to be bootable). The trick is now to make a dd image of the complete hd (example: dd if=/dev/hda of=~/hd.dd).

This is now bootable as in the previous floppy example, but since it takes the whole hd, the dd image can be large, luckily memdisk supports gzip. So gzip -9 your-dd-image.dd does a neat trick here.

Copy the whole cd content to hdd using your cd/dvd
Create new partitions

Please change grub configuration and beware of the double config grub.conf and menu.lst. Please remove one and set a link to the other file.

Edit /etc/fstab equal to your configuration.

Problems with hardened Kernel and chroot deny
If you want to chroot to a copied environment where the CONFIG_GRKERNSEC_CHROOT is enabled you must use the cd grub and change the root(cd) kernel(cd) initrd(cd) setting to from (cd) to (hdx,y).

Now you can install the grub environment.

COMPLETE BUILD SCRIPT
I had problems running these scripts: mounting/umounting different parts (sys, proc, ...), so I change to a plain text method. This way, reader can understand step by step what's happening and can check what's wrong.

build script
To help automate these steps, create a simple 'build' script inside ${LIVECD}

This builds the complete LiveCD from scratch at once. Be careful: it replaces menu.lst with grub.conf (so we do not forget!)

Create the build script in ${LIVECD} and give it chmod +x (call it build.sh for example).

shellscript.sh

Emerging fuse and dmraid makes sure we do not forget to build these modules after updating the kernel.

Make both executable, and be sure to read the warning message. If you do not check if it has successfully chrooted to the ~/target/files/source directory and you run the file, it might DESTROY your whole system!

Squashing a large source directory takes some time, so be patient. Once done, just burn your ISO image, boot it, and enjoy :)

Work script
A basic work script which enables you to easily work in the chroot, a more advanced automated script is below (simply call it work.sh (or similar) and give it chmod +x).

Here it is:

Do not forget, after running this script when you get in the environment, to manually run;

Comment: To automate this step just remove the --login part and add the following to source/root/.bash_rc

This is a more advanced version of the workscript, it only mounts the first console and only umounts after all consoles have left.

Work script