And Converting a 4-Way ZFS Mirror Into a RAIDZ Array Via a Network Reinstall

I’ve had great luck over the last few years renting SoYouStart’s super-cheap, but high powered dedicated servers. The prices are phenomenal, and the sale prices are even better. The SYS web UI is outstanding, especially if you are moving up from one of the heard of half-baked WHMCS installs. Everything is wonderfully automated. Their technical support is… Well their prices are really great, and better servers come with IPMI. Love it.

I recent ordered a pair of new servers that were a little different that the one’s I’d previously gotten. All my previous machines had been running FreeBSD on ZFS mirrors, but these new machines came with four big drives each. However upon delivery, I noticed that they’d been provisioned as a four-way ZFS mirror. That’s four disks all mirroring each other, quadruple redundant, with the total usable space of a little less than one disk. Sad trombone.

What I actually wanted was one RAIDZ array, and one RAID10 array.

The Dark Ages

So I tried to reinstall through their web UI with a custom layout, which made it easy to set the swap to something more reasonable than their stock offerings but the only options were to install on a single drive or an N-way mirror. No-go.

Then I checked out their “install template” options. They don’t seem to be documented anywhere, and they don’t really give you any low-level ZFS options, just zvol choices. Fail.

But I have IPMI, so I can do anything I want, right? I saved their installed network settings (important!) and launched their IPMI java console. Off the bat, it was a lot easier to use than other providers. I have had to jump through so many dumb hoops to get IPMI actually working on a server where it was sold as a fearture, but treated like a annoying inconvenience by the data center. And the console *actually* worked in OSX! Oh my!

Except it didn’t because it doesn’t support media redirection (mounting CDs over the network) on OSX. But I have run into this before, so I booted VirtualBox on my MacBook, installed Windows, and connected to their IPMI console. Nope, need to install java first. I always forget that. So many swears. Ok, with that installed I could use my OSX laptop, to run a Windows VM, to run a KVM console in a Java virtual machine, to install FreeBSD on a server in another country. So may yaks! But that got it working!

Until it didn’t. The boot disk took *forever* to load, and then seemed to hang after it loaded “ums0” the virtual keyboard/mouse device. It just hung there forever. I tried different ISOs and different versions of FreeBSD, but they all hung somewhere late in the boot. I tried the memstick mini image but that seemed to reliably crash the java console. Disaster. Now what?

Im going to skip ahead through a lot of hair pulling and failed schemes, and I’m not even going to mention trying to break three disks out of the mirror on a running system and build a RAIDZ with a fake memory disk in the fourth slot. Oof!

The Enlightenment

I finally got on Freenode #FreeBSD and begged for help. RhodiumToad pointed out that so late in the boot, when the machine was already in multi-user, it was a very strange place to hang, a he felt that it was booting fine, but the console output was getting sent to the wrong console. I poked around at his direction, with a very slow boot in between each try, and he finally suggested that we should disable serial output during boot, and that’s how we fixed it!

I booted from the v12.1 bootonly.iso, and then at the boot menu, I selected option #3, “Escape to the loader prompt”. That drops you into a command line “OK” prompt. I could see that the serial console was enabled with:

> show boot_serial
> YES

So I disabled it, and then continued the boot with:

> set boot_serial=NO
> boot

And I was in the installer! SYS actually had two console options. In addition to the Java IPMI console, there is an option for a browser-based serial console. At some point during the boot, FreeBSD switched the console to that and I could no longer interact with the Java console. I couldn’t simply use the serial console because it doesn’t support media redirection.