HOWTO: FreeBSD 8.2-RELEASE HVM guest on Xen 4 / Debian Squeeze

You may wish to read my update as it contains some important information about disk partitioning and PF performance.

Recently, I upgraded my server ashpool to Debian Squeeze, from Lenny. Along with this upgrade, I decided to start using the Debian-packaged Xen 4 (I’ve had a binary install of Xen 3.2 on it for a very long time).

A drive failure (thankfully in a RAID array) gave me the perfect chance to replace the drive, do a dist-upgrade, install Xen 4, and see if an HVM guest would be viable for replacing the venerable discordians.net hardware.

The upgrade was a bit of a chore, and involved a lot of grub2 woes that I won’t bore you with. Suffice to say I spent about an hour getting the bootloader on ashpool back into a serviceable state; I ended up just recreating my /boot partition to give GRUB it’s precious bios boot partition (my boot partition on Ashpool is >2TB, requiring the usage of GPT) that it apparently can’t live without.

Once Ashpool was online again and moderately happy (and outputting correctly to serial), I started to look into HVM guests. I had never used HVM before, and didn’t find much documentation related specifically to FreeBSD HVM on amd64, so I figured in case someone else out there is trying to do the same thing, I’d document what I did to make the process quick and painless.

Part of ashpool’s upgrade was motivated by a desire to use the latest Xen and also, as I said, experiment with HVM. However, another motivation is that I have wanted to consolidate services on one machine for a while, to save cost. However,I had no interest whatsoever in migrating away from FreeBSD, which I’ve been using on eris.discordians.net since 4.6. Until recently, Xen+FreeBSD was a little dicey, so now seemed like a great time to revisit it, especially since 8.2R came out a few days ago, and includes a number of HVM improvements ported from CURRENT. amd64 still only works as an HVM, whereas 32-bit works as a PV dom0, so HVM it is.

We begin by creating some drives for your new FreeBSD instance. With Xen 4.0.1, you can use LVM devices directly with FreeBSD (this didn’t seem to work with 3.x versions):

ashpool:/var/log/xen# lvcreate -L 250G /dev/Xen -n eris.discordians.net-disk
ashpool:/var/log/xen# lvcreate -L 16G /dev/Xen -n eris.discordians.net-swap
ashpool:/var/log/xen# lvcreate -L 750G /dev/Xen2 -n eris.discordians.net-data

I made 16GB of swap, a 250GB “root” drive, and a 750GB volume for user data, which pretty much mirrors what is in production.

Next, I created my HVM config file (see below). I didn’t want to mess around with building a FreeBSD system from scratch: this is HVM, so I wanted to just use an installation CD. I thought I could just download the ISO and use file:/ to connect it to the guest, however, it turns out that Xen doesn’t support attaching files directly to a guest device. Oh well.

To get around this, you’ll just have to create a loopback device for the ISO image. I found this on the Debian Wiki entry on Xen. If you’re getting a HVM BIOS message about CDROM boot errors, this is generally the issue, and you can fix it by creating and using a loopback device for the ISO image instead, using losetup.

Here’s what I did to create a loopback device using our FreeBSD 8.2 ISO:

ashpool:~# losetup -f /root/FreeBSD-8.2-RELEASE-amd64-disc1.iso
ashpool:~# losetup /dev/loop0
/dev/loop0: [0802]:504285 (/root/FreeBSD-8.2-RELEASE-amd64-disc1.iso)

Once you have the loop device, you can update your disk entry in your Xen config, to include an entry for our “cdrom drive”:

 phy:/dev/loop0,ioemu:hdd:cdrom,r

After that, because it was late and I was getting tired, I started getting some really weird errors:

ashpool:~# xm create /etc/xen/eris4.discordians.net.cfg
Using config file "/etc/xen/eris4.discordians.net.cfg".
Error: Invalid mode

It turns out that my ‘disk’ entry in my configuration file was missing some commas. :/ If you get that error, be sure to check that your disk and vif entries are completely correct. Oops.

Anyways, once that was fixed, I was able to boot the HVM guest, and connect to it using vnc. I just forwarded VNC via SSH to localhost, so I didn’t use a password (you can set one by giving vncpasswd a value in your Xen config file). Installation then proceeded completely normally, and the networking came up without any issues whatsoever, which was a pleasant surprise. One issue which i haven’t figured out is there is about a 30-45 second delay at boot. I haven’t yet had time to figure out what, exactly, is causing the delay, but once the system is booted, everything runs normally.

Here is the configuration I used, in full:

Xen configuration

name = 'eris4'

kernel = "/usr/lib/xen-4.0/boot/hvmloader"
builder = 'hvm'

memory = 16384
shadow_memory = 8
vcpus = 4

device_model = '/usr/lib/xen-4.0/bin/qemu-dm'

disk = [
     'phy:/dev/loop0,ioemu:hdd:cdrom,r',
     'phy:/dev/Xen/eris.discordians.net-disk,ioemu:hda,w',
     'phy:/dev/Xen/eris.discordians.net-swap,ioemu:hdb,w'
]

vif = ['ip=133.73.133.7,mac=00:00:DE:AD:BE:EF,bridge=eth0' ]

boot='dc'
serial = 'pty'

sdl=0
vnc=1
vnclisten='127.0.0.1'
vncconsole=1
stdvga=0

on_poweroff = 'destroy'
on_reboot = 'restart'
on_crash = 'restart'

Change certain values as appropriate, obviously.

Once the installation is complete, you can remove the loop0 entry from your configuration. I also changed the boot order (boot= in the Xen config) to “cd” to boot from the hard disk first.

I did this and proceeded to restart the 4th iteration of Eris:

ashpool:~# xm create /etc/xen/eris4.discordians.net.cfg
Using config file "/etc/xen/eris4.discordians.net.cfg".
Started domain eris4 (id=33)

I was then rewarded with the following:

FreeBSD/amd64 (eris4.discordians.net) (ttyv0)

login:

….And the peasants rejoiced.

There are a few things left, however. One of those things is rebuilding the kernel on the new HVM guest. FreeBSD now ships with a kernel configuration (XENHVM) which, among other things, will build paravirtual drivers, which will increase network and disk performance. We also want to make sure that xm console works correctly from our hypervisor, which it won’t at first.

You can build a new Xen-aware kernel easily, assuming you have a full source tree installed at /usr/src on the FreeBSD guest:

cd /usr/src
make buildkernel KERNCONF=XENHVM
make installkernel KERNCONF=XENHVM

I generally end up customizing my kernel for a number of reasons (I usually just remove things like wireless support). Here’s my config file:

eris4 kernel config file

cpu			HAMMER
ident		ERIS4-XENHVM

options 	NO_ADAPTIVE_MUTEXES
options 	NO_ADAPTIVE_RWLOCKS

options		XENHVM
device	  	xenpci

makeoptions	DEBUG=-g	

options 	SCHED_ULE		
options 	PREEMPTION	
options 	INET			
options 	INET6			
options 	SCTP			
options 	FFS			
options 	SOFTUPDATES		
options 	UFS_ACL			
options 	UFS_DIRHASH		
options 	UFS_GJOURNAL		
options 	MD_ROOT			
options 	NFSCLIENT		
options 	NFSSERVER		
options 	NFSLOCKD		
options 	NFS_ROOT		
options 	MSDOSFS			
options 	CD9660			
options 	PROCFS			
options 	PSEUDOFS		
options 	GEOM_PART_GPT		
options 	GEOM_LABEL		
options 	COMPAT_43TTY		
options 	COMPAT_FREEBSD32	
options 	COMPAT_FREEBSD4		
options 	COMPAT_FREEBSD5		
options 	COMPAT_FREEBSD6		
options 	COMPAT_FREEBSD7		
options 	SCSI_DELAY=5000		
options 	KTRACE			
options 	STACK			
options 	SYSVSHM			
options 	SYSVMSG			
options 	SYSVSEM			
options 	P1003_1B_SEMAPHORES	
options 	_KPOSIX_PRIORITY_SCHEDULING 
options 	PRINTF_BUFR_SIZE=128	
options 	KBD_INSTALL_CDEV	
options 	HWPMC_HOOKS		
options 	AUDIT			
options 	MAC			
options 	FLOWTABLE		

options 	INCLUDE_CONFIG_FILE     

options 	KDB			
options 	KDB_TRACE		

options 	SMP			

device		cpufreq
device		acpi
device		pci

device		ata
device		atadisk		
device		ataraid		
device		atapicd		
device		atapifd		
device		atapist		
options 	ATA_STATIC_ID	

device		scbus
device		da

device		atkbdc		
device		atkbd		
device		psm		
device		kbdmux		
device		vga		
device		splash		

device		sc
device		agp		

device		uart		

device		miibus		
device		re		

device		loop		
device		random		
device		ether		
device		vlan		
device		tun		
device		pty		
device		md		
device		gif		
device		faith		
device		firmware	

device		bpf		

options 	USB_DEBUG	
device		uhci		
device		ohci		
device		ehci		
device		usb		

device		uhid		
device		ukbd		
device		ulpt		
device		umass		
device		ums		

When I rebooted with this kernel, however, I ran into the following:

xn0: Fatal error. Transitioning to Closing State
panic: do something smart

I looked into it and found this post about some issues with the guest acquiring its MAC address.. A workaround is to change it to a “normal” entry (e.g. remove type=iommu):

vif = ['ip=133.73.133.7,mac=00:00:DE:AD:BE:EF,bridge=eth0' ]

One further thing to note is that using the XENHVM kernel will change the network device from re0 to xn0, so make sure you update your rc.conf.

One last thing to fix is to get xm console to work correctly. This basically just requires setting up a standard serial console.

In /boot/loader.conf. add the following:

boot_multicons="YES"
boot_serial="YES"
comconsole_speed=9600
console="comconsole,vidconsole"

Then, edit /etc/ttys and activate ttyu0:

ttyu0	"/usr/libexec/getty std.9600"	dialup	on secure

Restart, and you’ll be able to see output (and login terminals) via both xm console and vncviewer.

So far, I’m very impressed with the performance and stability of the HVM guest. I plan on putting it through its paces with various stress tests, as I have read that occasionally lockups can be experienced under heavy load. I’ll make a followup post if I notice anything odd.