You may wish to read my update as it contains some important information about disk partitioning and PF performance.
Recently, I upgraded my server ashpool to Debian Squeeze, from Lenny. Along with this upgrade, I decided to start using the Debian-packaged Xen 4 (I’ve had a binary install of Xen 3.2 on it for a very long time).
A drive failure (thankfully in a RAID array) gave me the perfect chance to replace the drive, do a dist-upgrade, install Xen 4, and see if an HVM guest would be viable for replacing the venerable discordians.net hardware.
The upgrade was a bit of a chore, and involved a lot of grub2 woes that I won’t bore you with. Suffice to say I spent about an hour getting the bootloader on ashpool back into a serviceable state; I ended up just recreating my /boot partition to give GRUB it’s precious bios boot partition (my boot partition on Ashpool is >2TB, requiring the usage of GPT) that it apparently can’t live without.
Once Ashpool was online again and moderately happy (and outputting correctly to serial), I started to look into HVM guests. I had never used HVM before, and didn’t find much documentation related specifically to FreeBSD HVM on amd64, so I figured in case someone else out there is trying to do the same thing, I’d document what I did to make the process quick and painless.
Part of ashpool’s upgrade was motivated by a desire to use the latest Xen and also, as I said, experiment with HVM. However, another motivation is that I have wanted to consolidate services on one machine for a while, to save cost. However,I had no interest whatsoever in migrating away from FreeBSD, which I’ve been using on eris.discordians.net since 4.6. Until recently, Xen+FreeBSD was a little dicey, so now seemed like a great time to revisit it, especially since 8.2R came out a few days ago, and includes a number of HVM improvements ported from CURRENT. amd64 still only works as an HVM, whereas 32-bit works as a PV dom0, so HVM it is.
We begin by creating some drives for your new FreeBSD instance. With Xen 4.0.1, you can use LVM devices directly with FreeBSD (this didn’t seem to work with 3.x versions):
ashpool:/var/log/xen# lvcreate -L 250G /dev/Xen -n eris.discordians.net-disk ashpool:/var/log/xen# lvcreate -L 16G /dev/Xen -n eris.discordians.net-swap ashpool:/var/log/xen# lvcreate -L 750G /dev/Xen2 -n eris.discordians.net-data
I made 16GB of swap, a 250GB “root” drive, and a 750GB volume for user data, which pretty much mirrors what is in production.
Next, I created my HVM config file (see below). I didn’t want to mess around with building a FreeBSD system from scratch: this is HVM, so I wanted to just use an installation CD. I thought I could just download the ISO and use file:/ to connect it to the guest, however, it turns out that Xen doesn’t support attaching files directly to a guest device. Oh well.
To get around this, you’ll just have to create a loopback device for the ISO image. I found this on the Debian Wiki entry on Xen. If you’re getting a HVM BIOS message about CDROM boot errors, this is generally the issue, and you can fix it by creating and using a loopback device for the ISO image instead, using losetup.
Here’s what I did to create a loopback device using our FreeBSD 8.2 ISO:
ashpool:~# losetup -f /root/FreeBSD-8.2-RELEASE-amd64-disc1.iso ashpool:~# losetup /dev/loop0 /dev/loop0: :504285 (/root/FreeBSD-8.2-RELEASE-amd64-disc1.iso)
Once you have the loop device, you can update your disk entry in your Xen config, to include an entry for our “cdrom drive”:
After that, because it was late and I was getting tired, I started getting some really weird errors:
ashpool:~# xm create /etc/xen/eris4.discordians.net.cfg Using config file "/etc/xen/eris4.discordians.net.cfg". Error: Invalid mode
It turns out that my ‘disk’ entry in my configuration file was missing some commas. :/ If you get that error, be sure to check that your disk and vif entries are completely correct. Oops.
Anyways, once that was fixed, I was able to boot the HVM guest, and connect to it using vnc. I just forwarded VNC via SSH to localhost, so I didn’t use a password (you can set one by giving vncpasswd a value in your Xen config file). Installation then proceeded completely normally, and the networking came up without any issues whatsoever, which was a pleasant surprise. One issue which i haven’t figured out is there is about a 30-45 second delay at boot. I haven’t yet had time to figure out what, exactly, is causing the delay, but once the system is booted, everything runs normally.
Here is the configuration I used, in full:
name = 'eris4' kernel = "/usr/lib/xen-4.0/boot/hvmloader" builder = 'hvm' memory = 16384 shadow_memory = 8 vcpus = 4 device_model = '/usr/lib/xen-4.0/bin/qemu-dm' disk = [ 'phy:/dev/loop0,ioemu:hdd:cdrom,r', 'phy:/dev/Xen/eris.discordians.net-disk,ioemu:hda,w', 'phy:/dev/Xen/eris.discordians.net-swap,ioemu:hdb,w' ] vif = ['ip=220.127.116.11,mac=00:00:DE:AD:BE:EF,bridge=eth0' ] boot='dc' serial = 'pty' sdl=0 vnc=1 vnclisten='127.0.0.1' vncconsole=1 stdvga=0 on_poweroff = 'destroy' on_reboot = 'restart' on_crash = 'restart'
Change certain values as appropriate, obviously.
Once the installation is complete, you can remove the loop0 entry from your configuration. I also changed the boot order (boot= in the Xen config) to “cd” to boot from the hard disk first.
I did this and proceeded to restart the 4th iteration of Eris:
ashpool:~# xm create /etc/xen/eris4.discordians.net.cfg Using config file "/etc/xen/eris4.discordians.net.cfg". Started domain eris4 (id=33)
I was then rewarded with the following:
FreeBSD/amd64 (eris4.discordians.net) (ttyv0) login:
….And the peasants rejoiced.
There are a few things left, however. One of those things is rebuilding the kernel on the new HVM guest. FreeBSD now ships with a kernel configuration (XENHVM) which, among other things, will build paravirtual drivers, which will increase network and disk performance. We also want to make sure that xm console works correctly from our hypervisor, which it won’t at first.
You can build a new Xen-aware kernel easily, assuming you have a full source tree installed at /usr/src on the FreeBSD guest:
cd /usr/src make buildkernel KERNCONF=XENHVM make installkernel KERNCONF=XENHVM
I generally end up customizing my kernel for a number of reasons (I usually just remove things like wireless support). Here’s my config file:
eris4 kernel config file
cpu HAMMER ident ERIS4-XENHVM options NO_ADAPTIVE_MUTEXES options NO_ADAPTIVE_RWLOCKS options XENHVM device xenpci makeoptions DEBUG=-g options SCHED_ULE options PREEMPTION options INET options INET6 options SCTP options FFS options SOFTUPDATES options UFS_ACL options UFS_DIRHASH options UFS_GJOURNAL options MD_ROOT options NFSCLIENT options NFSSERVER options NFSLOCKD options NFS_ROOT options MSDOSFS options CD9660 options PROCFS options PSEUDOFS options GEOM_PART_GPT options GEOM_LABEL options COMPAT_43TTY options COMPAT_FREEBSD32 options COMPAT_FREEBSD4 options COMPAT_FREEBSD5 options COMPAT_FREEBSD6 options COMPAT_FREEBSD7 options SCSI_DELAY=5000 options KTRACE options STACK options SYSVSHM options SYSVMSG options SYSVSEM options P1003_1B_SEMAPHORES options _KPOSIX_PRIORITY_SCHEDULING options PRINTF_BUFR_SIZE=128 options KBD_INSTALL_CDEV options HWPMC_HOOKS options AUDIT options MAC options FLOWTABLE options INCLUDE_CONFIG_FILE options KDB options KDB_TRACE options SMP device cpufreq device acpi device pci device ata device atadisk device ataraid device atapicd device atapifd device atapist options ATA_STATIC_ID device scbus device da device atkbdc device atkbd device psm device kbdmux device vga device splash device sc device agp device uart device miibus device re device loop device random device ether device vlan device tun device pty device md device gif device faith device firmware device bpf options USB_DEBUG device uhci device ohci device ehci device usb device uhid device ukbd device ulpt device umass device ums
When I rebooted with this kernel, however, I ran into the following:
xn0: Fatal error. Transitioning to Closing State panic: do something smart
I looked into it and found this post about some issues with the guest acquiring its MAC address.. A workaround is to change it to a “normal” entry (e.g. remove type=iommu):
vif = ['ip=18.104.22.168,mac=00:00:DE:AD:BE:EF,bridge=eth0' ]
One further thing to note is that using the XENHVM kernel will change the network device from re0 to xn0, so make sure you update your rc.conf.
One last thing to fix is to get xm console to work correctly. This basically just requires setting up a standard serial console.
In /boot/loader.conf. add the following:
boot_multicons="YES" boot_serial="YES" comconsole_speed=9600 console="comconsole,vidconsole"
Then, edit /etc/ttys and activate ttyu0:
ttyu0 "/usr/libexec/getty std.9600" dialup on secure
Restart, and you’ll be able to see output (and login terminals) via both xm console and vncviewer.
So far, I’m very impressed with the performance and stability of the HVM guest. I plan on putting it through its paces with various stress tests, as I have read that occasionally lockups can be experienced under heavy load. I’ll make a followup post if I notice anything odd.