Xen4.1

This guide will walk you through building a Gentoo Linux system with kernel 3.2.1 + Xen 4.1 + drbd.

The setup procedure has changed a lot since prior versions so be wary of guides and documentation that isn't specifically for xen 4.1 + kernel 3:
 * You'll be installing gentoo-sources, not xen-sources -- Xen is part of the official kernel now
 * The xm command is no more, you'll be using xl instead and its syntax is identical -- the "XL toolstack" replaces xm and xend
 * Xen doesn't want to manage your network settings anymore. The distribution is supposed to handle it instead now... so if you want bridging, you must build bridges
 * See wiki.xen.org for details : MigrationGuideToXen4.1+

Prepare base system
emerge -DuNav system emerge -DuNav world
 * Follow section 1 here: http://forums.gentoo.org/viewtopic-t-840251.html
 * After base system is installed, make sure to update the system (with any other normal procedure you use)

Configure and Build Kernel
General setup ---> <*> Kernel .config support [*]  Enable access to .config through /proc/config.gz
 * emerge gentoo-sources
 * cd /usr/src/linux
 * make menuconfig
 * Assuming you are setting up Kernel 3.2.1-gentoo-r2 (other versions may vary):

Processor type and features ---> [*] Paravirtualized guest support ---> [*]  Xen guest support

Bus options (PCI etc.) ---> [*]  Xen PCI Frontend

[*] Networking support ---> Networking options ---> <*> 802.1d Ethernet Bridging [*] Network packet filtering framework (Netfilter) ---> [*] Advanced netfilter configuration [*] Bridged IP/ARP packets filtering

Device Drivers ---> [*] Block devices (NEW) --->   DRBD Distributed Replicated Block Device support < >  Xen virtual block device support <*>  Xen block-device backend driver

Device Drivers ---> [*] Network device support ---> < >  Xen network device frontend driver <*>  Xen backend network device

Device Drivers ---> Graphics support ---> -*- Support for frame buffer devices --- < >  Xen virtual frame buffer support

Device Drivers ---> Xen driver support ---> [*] Xen memory balloon driver (NEW) [*]  Scrub pages before returning them to system (NEW) <*> Xen /dev/xen/evtchn device (NEW) [*] Backend driver support (NEW) <*> Xen filesystem (NEW) [*]  Create compatibility mount point /proc/xen (NEW) [*] Create xen entries under /sys/hypervisor (NEW)  userspace grant access device driver (NEW)  User-space grant reference allocator driver (NEW)  xen platform pci device driver (NEW)

File systems ---> < > Ext3 journalling file system support <*> The Extended 4 (ext4) filesystem [*]  Use ext4 for ext2/ext3 file systems (NEW) [*]  Ext4 extended attributes (NEW)
 * Don't forget to add the required drivers for your networking and sata cards
 * If you use Raid, make sure to add the correct CONFIG_MD_RAID* entries to your config
 * make
 * make modules_install

If you experience issues with connecting to the console ensure the module "xen_gntdev" (userspace grant access device driver) is loaded before the xenconsoled process is started (you may have to restart it after loading the module).

Switch to Python 2.x
Xen-tools does not support Python 3.x so we'll need to switch the python version Available Python interpreters: [1]  python2.7 [2]  python3.1 * Available Python interpreters: [1]  python2.7 * [2]  python3.1
 * eselect python list
 * eselect python set 1
 * Make sure its set to 2.x: eselect python list

Install Xen

 * emerge xen xen-tools
 * rc-update add xenconsoled
 * rc-update add xenstored

If, during installation of xen / xen-tools you encounter an error about not being able to write to /var/lock, run `mkdir /run/lock` and try again. If, upon trying again, portage complains about collisions with files that no packages own, erase those directories and try, try again.

Setup booting
default 0 timeout 30
 * emerge grub
 * cd /boot
 * cp /usr/src/linux/arch/x86_64/boot/bzImage kernel-3.2.1
 * ln -s kernel-3.2.1 kernel
 * vim grub/grub.conf

title Gentoo Linux 3.2.1 + Xen 4.1 root (hd0,0) kernel /boot/xen.gz dom0_mem=512M module /boot/kernel root=/dev/md2 Probing devices to guess BIOS drives. This may take a long time.
 * grub

GNU GRUB version 0.97  (640K lower / 9216K upper memory)

[ Minimal BASH-like line editing is supported. For the first word, TAB lists possible command completions. Anywhere else TAB lists the possible completions of a device/filename. ]

grub> find /xen.gz (hd0,0)

grub> root (hd0,0) Filesystem type is ext2fs, partition type 0xfd

grub> setup (hd0) Checking if "/boot/grub/stage1" exists... yes Checking if "/boot/grub/stage2" exists... yes Checking if "/boot/grub/e2fs_stage1_5" exists... yes Running "embed /boot/grub/e2fs_stage1_5 (hd0)"... 17 sectors are embedded. succeeded Running "install /boot/grub/stage1 (hd0) (hd0)1+17 p (hd0,0)/boot/grub/stage2 /boot/grub/menu.lst"... succeeded Done.

grub> quit

Install and configure system tools
rc_sys="xen0"
 * emerge lvm2 bridge-utils vixie-cron syslog-ng logrotate
 * rc-update add vixie-cron
 * rc-update add syslog-ng
 * rc-update add sshd
 * rc-update add lvm2
 * vim /etc/rc.conf

Install DRBD
The kernel version for DRBD that comes with kernel 3.2.1 is drbd-8.3.11; So make sure you are installing a matching version number:
 * emerge drbd (or just emerge =sys-cluster/drbd-8.3.11-r1)
 * You can also check for the version number of the module: modinfo -k 3.2.1-gentoo-r2 drbd | grep ^version
 * rc-update add drbd

Configure Networking
bridge_xenbr0="eth0" config_xenbr0="192.168.1.123 netmask 255.255.255.0 brd 192.168.1.255" routes_xenbr0="default via 192.168.1.1"
 * cd /etc/init.d
 * ln -s net.lo net.xenbr0
 * vim /etc/conf.d/net

dns_servers="4.2.2.2"


 * rc-update add net.xenbr0 default


 * Do the same for your local lan for xenbr1/eth1 if you have one

Add bridge failed: Package not installed
[*] Networking support ---> Networking options ---> <*> 802.1d Ethernet Bridging [*] Network packet filtering framework (Netfilter) ---> <*> Ethernet Bridge tables (ebtables) support ---> <*> ebt: nat table support <*> ebt: 802.3 filter support
 * Make sure you have the net-misc/bridge-utils package installed
 * Make sure you enabled all the Networking support config options (CONFIG_BRIDGE, BRIDGE_NETFILTER) in your kernel (See Xen4.1), However, if you still get the error enable the following kernel options:
 * Note: I have not confirmed if the above makes a difference but I believe the problem is with BRIDGE_NETFILTER so this above may not be needed.

Don't forget
emerge mdadm mdadm --detail --scan >> /etc/mdadm.conf
 * To install mdadm and set your mdadm.conf if you use Software raid
 * To setup your /etc/fstab

Build a domU
h0:12345:respawn:/sbin/agetty 9600 hvc0 screen hvc0 rc_sys="xenU"
 * Unpack stage3 tarball and portage snapshot into a new ext3/4 filesystem
 * Chroot into system, install gentoo-sources, configure xen frontend drivers, make kernel
 * vim /etc/inittab
 * 1) add anywhere to create hvc0 console for Xen
 * vim /etc/securetty
 * 1) add anywhere
 * vim /etc/rc.conf
 * Copy kernel bzImage to dom0
 * Create domU config file to load kernel with the filesystem you just created

Autostart domU

 * Create symlinks to your domU configs within /etc/xen/auto
 * rc-update add xendomains

/etc/init.d/xendomains
After a fresh install of xen and xen-tools, you may find xendomains refusing to start with "ERROR: xendomains needs service(s) xend

xend is deprecated and no longer installed by default but xendomains still depends on it. Manually deleting xend from the need line near the top of /etc/init.d/xendomains seems to fix things. See Bug 395435

/etc/xen/xl.conf
If your domU start up OK but your networking to/from them is not working. You may need to provide full path for vifscript parameter e.g. vifscript="/etc/xen/scripts/vif-route" as opposed to vifscript="vif-route" (for route networking)  and vif-bridge (for bridge networking)

Resources

 * Gentoo Forums: setting up gentoo xen dom0 with multiple bridges
 * Gentoo Forums: Xen 4 and hardened paravirt domUs
 * Gentoo Forums: Kernel 3.0 and Xen
 * Xen Wiki: HostConfiguration/Networking
 * Gentoo Wiki: Paravirtualized_Gentoo_VMs_on_XenServer