KVM

KVM (for Kernel-based Virtual Machine) is a full virtualization solution for Linux on x86 or x86_64 (AKA amd64) hardware containing virtualization extensions (Intel VT or AMD-V). It consists of a loadable kernel module, kvm.ko, that provides the core virtualization infrastructure and a processor specific module, kvm-intel.ko or kvm-amd.ko which together provide for all that is needed to get virtualized CPU support.

KVM also requires a modified QEMU or QEMU 0.11 to provide for the remaining virtualized hardware (a virtual machine) including network, disk, and video adapters as well as block devices like hard drives, cdrom or floppies.

Be aware that not all Intel or AMD CPUs have virtualization extensions and some might need to have also a BIOS setting enabled in order to be able to use it. Refer to the KVM FAQ for more information.

This howto will explain how to install kvm on your Host machine, start the installation process of your Guest OS, configure the USB and network parts. We will show a practical example of two Guest OS with network enabled and able to talk to each other and to the Internet.

Hardware
Your CPU has to support full virtualization. Intel CPUs need to have the vmx capability, AMD CPUs require svm:

Packages
For this howto, we will need the following packages:
 * : needed for lsusb (see )
 * : needed for brctl (see )
 * : needed for tunctl (see )
 * : needed for masquerade (see )
 * : needed for masquerade (see )

Optional: app-emulation/virt-manager

TODO:Add information about KSM (samepage merging) which is included in kernel 2.6.32, see also the website

The virtual machine
To enable the KVM, you need the kvm virtual machine, and a driver. Most likely you will want to select the Intel or AMD specific driver depending on what CPU your host computer has, but if you want to run out-of-tree modules (USE flag modules dependent package app-emulation/kvm-kmod), you may get along without either.

Networking
If you want your virtual machine to access your network, you will need to support bridging, the TUN/TAP interfaces and VLAN.

Paravirtualization
If you want your virtual machine to have paravirtualization guest support, which can improve virtual machine performance in some scenarios, then you will need to support paravirtualization:

Kernel SamePage Merging (KSM)
Kernel SamePage Merging combines identical memory pages from multiple processes into one copy on write memory region, this provides KVM with a memory overcommit feature that is important to hypervisors for more efficient use of memory and an overcommit feature for memory allocation. If you need to run multiple virtual machines and memory is a limiting factor, then KSM is your ideal solution; alternatively, even in situations where memory is not a constraint KSM can provide better memory management and improve scalability:

You can now compile your kernel and install the modules:

Don't forget to copy your new kernel over, using the make command, it copies the kernel to /boot and renames any old kernel with the same name:

After booting your previously built kernel be sure to check

to return 1. Otherwise ksm won't be working.

To turn ksm on.

Configuring your USE flags (version: qemu-kvm-0.12.2-r1)
The ebuild has several configurable options that can be used to adjust it to your environment better. For more information about how to set the USE flags refer to HOWTO Use Portage Correctly. By default qemu-kvm has convenient USE flags enabled by its ebuild.


 * : Adds support for media-libs/alsa-lib (Advanced Linux Sound Architecture). This used to be enabled by default for "=app-emulation/kvm-45, the VNC server option available from the userspace application with the -vnc option is able to use TLS to encrypt the session established with the vnc client, this flag enables support for that functionality. If disabled vnc sessions are still possible but only using unencrypted channels.


 * : Adds support for PulseAudio sound server.


 * : Adds support for Simple Direct Layer (media library). If enabled (recommended), will use SDL to emulate a console, so you can interact with your virtual machine through an X session. If you are only interested on using kvm to run "headless" virtual machines then be sure to start them with either an emulated serial console going into a file (or /dev/null) or a vnc server console and disable this flag.


 * : Enable VDE-based networking. VDE is a package that provides software versions of physical networking devices (e.g. switches, cables, etc). In the same way that KVM allows you to virtualize a computer, VDE allows you to virtualize your networking gear.

Adding access to /dev/kvm to your user
The /dev/kvm device is owned by root but with read/write privileges for the kvm group, so in order to be able to access it you have to add your user to it:

Create a virtual disk image
To create a virtual disk image, use the qemu-img tool. The following example will create a 10G Copy On Write disk image:

This image uses the qemu specific qcow2 format. It supports encryption, compression, and snapshotting and grows dynamically. For better performance, the metadata will be preallocated, allowing easier growth, but increases the initial image size on the other hand.

Gentoo example
To install gentoo into the disk image using an livecd ISO:

You can look at the kvm man page to see that this command says:
 * run a virtual machine with:
 * an emulated hard disk drive using the file gentoo-i386.img
 * an emulated CDROM drive containing the emulated CDROM image livecd-i686-installer-2007.0.iso
 * and boot from device "d" (i.e., the emulated CDROM drive)

Windows example
As of 2.6.29 with 84/5 I am no longer able to start windows. I had to go down to 2.6.28. --Letharion With kvm-78 and kernel-2.6.24, nothing special was required to install Windows XP. The following command was used:

I have successfully installed Windows Vista Ultimate with kvm-84 and 2.6.28 like this:

Without the -m 512 Vista complained about too little available RAM.

I have successfully installed Windows XP with kvm-83 and 2.6.27 like this:

Once installation is complete, and you no longer need the disc, the command simply becomes

This ignores the CD, and requests boot from harddrive, instead of cd-device.

Kubuntu example
The latest ubuntu/kubuntu variant "Jaunty Jackelope" may have trouble with monitor drivers other than the default cirrus. You will probably also need to override the default boot options on its initial boot screen with "noacpi" (hit f6 for that option) and possibly "safe graphics" (f4 options) before booting the install. It also appeared to hang on its initial splash screen with -soundhw ac97 to enable audio.

This command line sets 2gb of memory, a tuntap interface for network, a name for the sdl window (kubuntu) so that more than one kvm sdl session can run without interference and the alternative grab sequence "ctrl-alt-shift" instead of "ctrl-alt". This is kvm-0.84 on a 2.6.28-r5 amd64 kernel where the tuntap and bridge interface are already plumbed up from init.d and conf.d/net settings:

After the initial installation, experiments showed that the es1370 virtual driver would allow kubuntu to boot and use audio. However the cirrus driver still appeared to be the only monitor option that would work. Thus vm display resolutions will be limited to 1024x768 until software updates from Canonical permit better driver support.

Fedora 10 example
The latest RedHat Fedora variant (at least until May 2009) does not have trouble booting the installer with acpi enabled, enhanced vga or ac97 driver support. This guest lives on tap1 alongside the kubuntu guest above:

After the install, Fedora will initially have troubles with the adjustment of the graphic display and system fonts when living in a vm. This is because it is using the new approach to Xorg which relies on the monitor's EDID information to calculate available display resolutions. You will need to use yum to install system-config-display (a sore point among the Fedora community that it was left out of the default install) in order to create the missing xorg.conf file and populate it with sane settings. You will also probably need to set a specific font resolution in kde's "system-settings -> appearances" to override the bad choices made for you.

USB support
KVM allows you to add USB support to your VM by "bridging" what the kernel of your Host sees to the Guest OS. To do so, you need to start qemu-kvm with the -usb flag on. Moreover, you will need to tell the Guest OS which usb device is being used for it. kvm (or more adequately qemu), can pass the USB VendorID and ProductID to the Guest OS in order for it to know that there is a USB attached. To know what VendorID and ProductID your USB device is, use the lsusb utility:

... Bus 001 Device 006: ID: 08ec:2039 M-Systems Flash Disk Pioneers ... In this example, the VendorID is "08ec" and the ProductID "2039". To launch qemu-kvm with USB support and this flash disk, you can use the following command:

If you have more than one USB device you would like to support, just add another "-usbdevice host::" to the command line.

Networking
There are a few ways to allow the Guest OS to talk with each other and the outside world.

Quick & Easy Networking
This is the most convenient method of networking. If you use Network Manager and just want to get your KVM virtual machine started and accessing the Internet, without doing all kinds of network setup, simply start kvm with the "-net user" switch:

Now your guest will have Internet access (assuming the host does), without setting up TUN,TAP,VDE, etc. However, you will not have local LAN access.

VDE Networking
An alternative way of networking all your virtual machines is to connect them to a virtual switch. This method is slightly easier than ethernet bridging.

Firstly you must make sure you compiled qemu with the USE flag. You can then start the virtual switch by running

Now when you launch your guest OS you can tell them to automatically connect to the virtual switch

Your guests will now be able to communicate with each other.

If you want your guests to be able to communicate with the outside world (e.g. your LAN or the internet) then you need to connect your physical interface (e.g. eth0) to the virtual switch as well. This can be done by running

(Note that vde_pcapplug is only available when you are using net-misc/vde-2.2.3 (currently masked) or above with the USE flag. One drawback with this vde_pcapplug approach is that the host and guest can't talk to each other directly due to limitations in linux kernel)

All incoming traffic on eth0 will now be copied to the virtual switch, and all traffic from the virtual switch will be sent out over eth0. This means your guests will appear as part of your physical LAN.

Possible Network Layouts
The principle is the following: you need to create an interface that will be the bridge between all the TUN/TAP interfaces (one per Guest OS) and the ethernet one of your Host OS. From there, you will be able to forward the traffic (using iptables) to the Internet.

The configuration described here after can be seen as follows:

Direct bridging (so guests use an IP address on the same subnet as the host): HOST +---+       |               |      KVM GUEST1 |              |   +--+        |  +--+     |   |              | LAN ---+--- eth0 |  +--+---+ nic0     |      KVM GUEST2 | | tap0+  |   |192.168.1.13  |   +--+ | | tap1+  |   +--+   |              | | +--+  |  |                      |              |        |     br0    +--+--+ nic0     | |192.168.1.12  |                      |192.168.1.14  |        +---+                      +--+

When use NAT/Masquerading (to hide the guests behind the host): HOST +---+       | 192.168.1.12  |      KVM GUEST1 LAN ---+ eth0     |   +--+ |     ^        |   |              |        |      |        |   |              |        |  +--+  +--+---+ nic0     |      KVM GUEST2 | | tap0+  |   |192.168.100.1 |   +--+ | | tap1+  |   +--+   |              | | +--+  |  |                      |              |        |     br0    +--+--+ nic0     | |192.168.100.254|                     |192.168.100.2 |        +---+                      +--+

Preparations
The most transparent option to allow your guests access to the internet is the "virtual hub". In this scheme, the bridge connects eth0 and your tuntap interfaces together, routing packets as if it were a real "old fashioned" hub (not a switch). The key to this approach is to make sure you have unique mac addresses on both the host's tuntap interface as well as the guest. The guest ip addresses are typically in the same subnet as the host, and they can ask for and receive a dhcp lease from the same dhcp server that the host might use if it used dhcp. That is because all arp traffic and other broadcasts are passed through the bridge between the eth0 interface and the guest taps. The guests can use the same default gateway as the host because of this transparent passage.

The following snippet from an /etc/conf.d/net file shows the setup of a bridge between eth0 and two tap devices for guests. Note that the dependency for eth0 is left out of the br0 config since it is always started earlier on this particular system.

Direct bridging:

Using NAT/Masquerading:

This is essentially the conf.d/net setup for the Fedora10 and kubuntu guest examples above. Note that the bridge br0 uses a different subnet otherwise the routing table will be ambiguous. Note also that their nic definitions on the kvm command line use different mac addresses than what is set for their taps. If the same mac address had been used on both sides, the arp queries for address resolution would not work. This conf.d/net setup is also why the kvm command line says not to do anything about interface startup or takedown.

You need to create the appropriate net.br0, net.tap0 and net.tap1 initscripts. You can use ln to do this:

You also need to add net.br0 to the default runlevel:

Finally you need to make some changes to /etc/sysctl.conf and add a new init-script. The changes in /etc/sysctl.conf are to prevent the traffic from the guests to be sent to iptables to be filtered. If you want to filter the traffic from/to the guests, you can keep the file unchanged but you will have to add the correct rules to iptables. The addition needed in /etc/sysctl.conf is ...

The new init-script is to allow br0 to forward the traffic from your guests. This can unfortunately not be added to the /etc/sysctl.conf because the br0 interface does not exist when those changes are applied. The best way I have found to fix this is to add a file called /etc/init.d/bridge_forward with the following content ...

Don't forget to add the script into default run level

If you have net.br0 configured up with dhcpcd. You may need to delete dhcpcd service from default run level to prevent ambiguous route problem.

The alternative to the virtual hub is to enable masquerade on your machine:

The disadvantage of this approach is that the initial inbound connections to your guests will not work without dnat rules. This can become quite involved and generally will get you to look at using shorewall for the management.

Start your Guest OS
This section will show how to launch two Guest OS with kvm and the setting up. If you need the USB support, just append the USB options as shown earlier to the commands here after.

Launch KVM Guest1 with network enabled
Now that the requirements are included in the kernel or the modules compiled, we need to load the modules:

Let us create the bridge interface br0 and bring it up:

The next step is to create the TAP interface:

Now that we have the TAP interface and the bridge one, we need to link them:

And finally, we bring tap0 up with the "promisc" mode:

Ok, you should now have br0 and tap0 appearing with ifconfig:

br0      Link encap:Ethernet  HWaddr 00:00:00:00:00:00 inet addr:192.168.100.254 Bcast:192.168.1.255  Mask:255.255.255.0 UP BROADCAST RUNNING MULTICAST MTU:1500  Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:6469 (6.3 KiB) TX bytes:410 (410.0 B)

eth0     Link encap:Ethernet  HWaddr  inet addr:192.168.1.5 Bcast:0.0.0.0  Mask:255.255.255.255 UP BROADCAST MULTICAST MTU:1500  Metric:1 RX packets:0 errors:0 dropped:0 overruns:0 frame:0 TX packets:0 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:0 (0.0 B) TX bytes:0 (0.0 B)          Interrupt:17

lo       Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436  Metric:1 RX packets:117562 errors:0 dropped:0 overruns:0 frame:0 TX packets:117562 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:8250309 (7.8 MiB) TX bytes:8250309 (7.8 MiB)

tap0    Link encap:Ethernet  HWaddr 00:00:00:00:00:00 UP BROADCAST RUNNING PROMISC MULTICAST MTU:1500  Metric:1 RX packets:81 errors:0 dropped:0 overruns:0 frame:0 TX packets:6 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:500 RX bytes:7669 (7.4 KiB) TX bytes:410 (410.0 B)

Now, you can boot your VM with the following parameters:

The two "-net" switches have a different meaning but are both needed:
 * nic,macaddr=00:00:00:00:00:00: we specify the options related to the network interface of the Guest OS. Please note that the macaddr needs to be specified
 * tap,ifname=tap0,script=no</tt>: we specify how kvm should work with the network on the Host side. The "script=no" means we will not use the given scripts in /etc/kvm/kvm.ifup as we already tuned everything before.

When using NAT, do not forget to configure KVM Guest1 (192.168.100.1) with the br0 interface (192.168.100.254) as the default gateway. (And you probably also wanna take the DNS-settings from /etc/resolv.conf --Letharion)

When using bridging, set KVM Guest1 (192.168.100.1) with the same default gateway as the host.

Launch KVM Guest2 with network enabled
What needs to be done as we already loaded the modules and the bridge interface, we just need to create the TAP/TUN interface and link it to br0:

Now, you can boot your second VM with the following parameters:

When using NAT, do not forget to configure KVM Guest2 (192.168.100.2) with the br0 interface (192.168.100.254) as the default gateway.

When using bridging, do not forget to configure KVM Guest2 (192.168.100.2) with the same default gateway as the host.

Now, your two Guest OS should be able to ping each other.

Script to ease the configuration
If you want to ease the process of configuring your machine (load the modules, create the bridge interface, the TUN/TAP, ...), here is an init.d script:

Modprobe kvm-amd gives general protection fault
If CONFIG_KVM=n, kvm-kmod cannot be built, while if CONFIG_KVM=y, there is a possible conflict between in-kernel kvm and out-of-tree modules. Personally, I get general protection fault trying

So, it should be CONFIG_KVM=m.

My x86 Gentoo Guest dies with a kernel panic
KVM doesn't emulate the MMR registers needed for performance metrics and therefore it will generate a General Protection Fault if they were used. A workaround was added to the Linux kernel 2.6.22.5 and therefore upgrading the kernel in your guest will fix this problem. If you can't upgrade your kernel try using the nolapic or nmi_watchdog=0 boot parameters instead.

But I can't boot my guest anymore
If you are using the USE-Flag qemu then you can use qemu + kqemu instead to try to fix any problems with the guest that affected kvm, if that doesn't work try using kvm -no-kvm or qemu -no-kqemu as a last resort.

modprobe kvm modules error
must not been set as No Forced Preemption (Server)

When you use AMD CPU and get such an output error in: ... kvm: Unknown symbol intel_iommu_domain_alloc kvm: Unknown symbol intel_iommu_detach_dev kvm: Unknown symbol intel_iommu_page_mapping kvm: Unknown symbol intel_iommu_context_mapping kvm: Unknown symbol intel_iommu_iova_to_pfn kvm: Unknown symbol intel_iommu_domain_exit ... then turn off the experimental modules could help:

What about when you're using an Intel CPU, these drivers are off, and you STILL get these same errors?? Or is kvm-81 just broken?

A bug on KVM's tracker indicated that CONFIG_DMAR may be to blame. ( http://sourceforge.net/tracker/?func=detail&atid=893831&aid=2405145&group_id=180599 )

Can I boot an Operating System (Windows, Linux, etc) residing on a partition instead of an image?
The key here is you must specify the hard disk (or lvm) (e.g. /dev/sda) that the OS resides on and not the specific root partition (e.g. /dev/sda3). If you boot multiple operating systems through grub normally, you should be greeted with a grub screen where you can select which OS you would like to boot.

External References

 * Qemu (kvm) internal network setup
 * http://q.deltaquadrant.org/index.php/KVM
 * Kernel Based Virtual Machine Homepage
 * Qemu Homepage
 * HOWTO: Setting up QEMU on Ubuntu with TUN/TAP and NAT
 * KVM / QEMU Easy Routed Networking (TAP interface without bridge)
 * HOWTO: Using KVM with KSM on archlinux

KVM