User:Owen/HOWTO Boot From SAN

A Few Things
This guide will show you how to install Gentoo directly to a LUN on your SAN and boot from this LUN. I am using a QLogic HBA and I have a Sun StorageTek 6140 disk array, but I'm sure much of this could be applied to a different setup.

Assumptions:
 * You already have the initiator/host configured on the array controller
 * 1 LUN/volume is mapped to this host (the one we'll install Gentoo on)

EMC Support:

Install CD
Enable the QLogic BIOS option (CTRL + Q to enter controller configuration at boot time). Also, in the configuration menu, set the LUN you would like to boot. Set the HBA controller card to the first boot device in your system BIOS. Boot your Gentoo install CD. Configure your network according to the Gentoo Handbook. I am using a 2006.1 install CD, but the firmware doesn't appear to be included: qla2xxx driver Not a problem, we just need to grab the binary blob and put it into /lib/firmware like this:

Of course you'll need to make sure you get the appropriate file for your card. Now that you have the firmware and its accessible by the qla2xxx driver, simply unload and load the module:

Now our volume should be detected: qla2xxx driver I ran into a strange problem when I first started setting this up; I couldn't access every other LUN I added to the host. I would get block error messages. To fix this, I changed the initiator host type to Solaris. I was using host type Linux, which seems to be the obvious choice, but I had that problem. Using host type Solaris seems to work fine.

So, now that we have our SCSI disk (/dev/sda) you can follow the Gentoo Handbook for setting this up (just like any other disk you install Gentoo on). The device node may be different for you, especially if you have other SCSI/RAID disks in your system. Continue the install procedure following the Gentoo Handbook until you get to setting up the kernel.

Set up Driver
Now its time to configure the kernel and HBA driver, the fun stuff. =) We have a couple options here, you can use QLogic's driver (my personal choice) or you can use the qla2xxx driver that comes with the Linux kernel.

If you use QLogic's proprietary driver, you get MPIO (failover) at the driver level and you won't have to install another MPIO solution (such as multipath-tools, EMC's PowerPath, Sun's RDAC driver, etc.). The firmware is also included at compile time, so you won't have to mess around with an external firmware file.

If you use the qla2xxx driver thats included in the kernel source, it will be updated with each kernel revision. You will still have to worry about external firmware for this driver though.

You'll still end having to create an initrd image when you want to update the driver with either choice. Oh, one more thing, no genkernel ladies! All of these directions assume you do manual configuration and configure a mostly monolithic kernel.

Option A: Using QLogic's driver
Configure the kernel manually like usual making sure you enable loading/unloading modules, RAM disk support and initramfs/initrd (loopback device is also helpful), but DO NOT select an HBA driver. Compile and install the kernel/modules like normal. Now, you need to get the driver from QLogic's website and put it into /usr/src. The driver I obtained is qla2xxx-v8.01.07-1-dist.tgz. Make sure you get the newest driver; the previous version of this driver would not work with 2.6.19 kernels. Now lets get to it:

I had to change 2 lines in the extras/build.sh script:

For this new kernel version, I also had to change some lines in the source files. See this note from Linus: You probably just have to change the file that is required for your HBA, but it doesn't hurt to change them all:

Change the "pci_module_init" in each file to "pci_register_driver" otherwise you will get a 'Unknown symbol: pci_module_init' when attempting to insert the module.

You should then get something like this:

I think the error at the end comes from depmod. If you select other drivers or options (that the QLogic driver may depend on) as modules, you should probably resolve the error so depmod finds your module dependencies correctly. Also, if you get additional errors, diagnose and fix whats wrong then do a 'extras/build.sh clean' and 'extras/build.sh install' again.

Now continue configuring your system and installing the necessary system tools according to the Gentoo Handbook.

We need mkinitrd:

Its time to create the initrd image:

I'm not sure if this next step is totally necessary, but it gives the modules some time and makes the output a little neater on the screen. I also like to make sure the linuxrc script looks kosher.

I just edit the /mnt/linuxrc file and add 3 sleep commands to it after each insmod (there are 3 modules), so my linuxrc now looks like this:

Again, its probably not necessary to do that, but I did anyways. =) Now put it back to normal:

Make sure you make the initrd mount point (for when pivot_root moves root):

We need to make sure the kernel knows what our root device is:

If it doesn't give you the correct root device you're going to be using, set it with rdev. I've found that initrd basically ignores the 'root=' line in grub.conf or at least it doesn't like it when they're not the same. I had strange problems with this. Anyone got any ideas? Anyone else experience the same weirdness?

Option B: Using Linux kernel driver
Configure the kernel manually like usual making sure you enable loading/unloading modules, RAM disk support and initramfs/initrd (loopback device is also helpful), and select the QLA2XXX HBA driver to compile as a module (its in SCSI low-level drivers). When you select the driver to be a module, make sure you take note if it selects any other stuff. If it selects dependencies as modules, be sure to change them to built-in. Compile and install the kernel/modules like normal.

Now continue configuring your system and installing the necessary system tools according to the Gentoo Handbook.

We need mkinitrd:

Next we'll need to create the initrd image:

We need to add the firmware file to our initrd image. I also add a sleep after the insmod, probably not needed, but I do it anyways.

I just edit the /mnt/linuxrc file and add the sleep command to it after the insmod, so my linuxrc now looks like this:

Next we can add the firmware file to the initrd image:

Now put it back to normal:

Make sure you make the initrd mount point (for when pivot_root moves root):

We need to make sure the kernel knows what our root device is:

I had strange problems with initrd if rdev wasn't set right.

Configure Bootloader
We will now configure the bootloader:

You'll notice I didn't set a 'root='; I actually had problems with initrd when I would set 'root=' in grub.conf. If you're kernel has the correct rdev set (as mentioned in step 2) you don't really need one of these anyhow.

With your grub.conf file set, lets move on:

You might have to switch some things around; grub-install displays your device nodes along with the BIOS disk (hd0, hd1, etc.). Make sure this corresponds to what you put in grub.conf. If you followed the directions in step 1 (enabled your QLogic BIOS, set boot LUN and set the HBA controller to the first boot device in system BIOS) then it should be hd0.

Reboot according to the Gentoo Handbook and you're done!