So I recently upgraded my server (the one this site runs on) from an old SheevaPlug running Debian Linux to a brand new Haswell Pentium running FreeBSD 10.0. I switched for a variety of reasons. Long story short, the SheevaPlug’s hardware was limiting me too much, and FreeBSD’s awesome ZFS file system as well as it’s Jails feature.
So lets get into this story. Or I should say, the back story first.
Hard drive musical chairs. I tried explaining this to a few friends, and it’s hard to wrap your head around all the drive swapping. So let’s explain the drive arrangement when I started.
I had a 2TB USB drive. I removed it from the USB enclosure and installed FreeBSD onto it. Formatted the root as ZFS. I then moved most of my data from my old 4TB drive over to this new system. The 4TB was in a USB box initially too. Once I got all the data moved over I removed that 4TB drive from it’s box, and put it in the new server. I made it a ZFS mirror of the 2TB, so I was only able to have a 2TB pool. Fair enough, tight fit for the time being, but I planned to swap out the 2TB later on. Both of those drives were Western Digital.
Next, I had a really old 1TB Hitachi USB hard drive. And a Seagate 1TB that I had warrantied about 2 years ago when it failed prematurely. I set those both up as a mirrored 1TB pool. Worked out great, and I had the system up and running for 2 whole months.
Last weekend comes along and I check the status of my zpools and I see I’ve lost a drive. Checking the logs, I see it failed 3 days earlier (Tuesday) at 5:15pm. Love that, it just kept humming along without any real problems. Thanks to each drive being completely different, I had no problem determining that it was the damn Seagate. Failed for the SECOND time in it’s life. This time the warranty had expired. Grrr. Needless to say I did not order Seagate replacement drives, despite being a diehard Seagate fan for years.
Now the story really begins.. I ordered a 2TB and a 4TB Western Digital NAS drives. I want to end up with a 4TB mirrored pool, and a 2TB mirrored pool (vs the 2TB and 1TB pools I had started with).
The main drive is the old 2TB. It’s mirror wasn’t partitioned to be bootable, so I need to delete it from the pool, partition it properly to be bootable. So delete mirror, create GPT partition table, create boot, zfs, and swap partitions, make it a mirror again, and “resilver” the old 2TB back onto it, which should take quite awhile (6 hours if I remember right). Then I need to shut down, set the bios to boot off the old 4TB and confirm that I actually made it bootable.
Assuming that goes well, I can now delete the Hitachi 1TB from the “small” pool, and repartition the old 2TB to a single ZFS partition, and make it a mirror of the new 2TB drive. That should automatically expand that pool from being 1TB to 2TB when it’s done.
Now I can shut down, remove the 1TB Hitachi drive and add the new 4TB WD NAS drive. Boot back up, set up the 3 partitions to make it properly bootable. I should have a good idea of how to do so at this point. Add it to the zroot pool, resilver, which should take quite awhile again. And again, the zroot pool should automatically expand from 2TB to 4TB once the second drive is added to it (if it didn’t already when I deleted the 2TB from it’s mirror list).
If all goes well, the machine will have 6TB of redundant data, vs the 3TB it had initially. :) It was VERY full with only 3TB of drives.