Previously I was using an unRAID setup for my home server, but lately I’ve been hearing a lot of ZFS and wonderful things about it. I decide to migrate my home server to FreeBSD+ZFS (Due to a license issue, Linux still hasn’t fully embraced ZFS yet.)
Here are my criteria for a home server:
Data integrity and reliability. It must offer ways to backup important data and ensure they are safe even when (some) hardware fails.
Small foot-print and power efficient, with little or no human intervention. The system should spin down the disk, and go into sleep mode (suspend itself either to memory or to disk) when not used, and can be easily woken up when needed, e.g., via WOL (wake-on-lan).
Media streaming to PS3, TV, Roku Radio, etc.
With some customization, my previous unRAID setup has almost met all the above requirements: entire system boots off a small USB memstick; a mirror setup of two 1TB SATA hard disk keeps all my data safe; media streaming is handled by add-on programs, etc., except that its backup solution is no more than file copying.
This is where ZFS really shines: it allows direct access to historical snapshot within the file system. ZFS snapshots are also very cheap to create since unmodified historical files are only stored once and shared among snapshots. So it makes a very convenient incremental backup solution.
As a side node, I’ve been using Nilfs2, a log based file system, on my laptop. While its automatic snapshot is tremendously helpful on occasions and I was also using it as an incremental backup solution, Nilfs2 is still rather experimental. I’ve had at least 3 bad crashes, and still have not recovered my files from a most recent disaster (I still hope I can!).
Long story short, this article documents what I did to migrate my home server to FreeBSD+ZFS. This is largely a note to myself, but if you are reading it, I assume you have at least some hands-on, command-line experience with Linux or BSD.
To reduce the impact to my existing setup, I first did a base installation of FreeBSD (latest is 8.1 at the time of this writing) onto a portable hard disk plugged into the server via a USB cable. I chose the AMD64 flavor of FreeBSD because ZFS seems to like 64-bit OS better. My server doesn’t have a CDROM drive, so I downloaded the amd64 memstick distro, dumped it to a USB memstick, booted from it and installed to the portable HD. Everything went as expected, and I made sure that I was only formatting and installing to my portable HD without touching the two 1TB internal hard disks where I kept my data.
After a reboot (from the portable HD), I was in FreeBSD.
One of the reasons made me choose unRAID previously was that its disks are individually mountable as ordinary ReiserFS. The usual software RAID implementations in Linux and FreeBSD are incompatible, and disks used by one cannot be handled by the other. This is where unRAID really makes data migration easier since I wouldn’t need the old server to be around when doing the new one.
I had two 1TB drive setup as a mirrored system under unRAID, one data and the other parity. Each can be mounted under Linux. But now I discovered that they cannot be mounted under FreeBSD. Even though FreeBSD has read-only support for ReiserFS, it still gave mounting error saying the file system was not recognized.
I googled a bit on this problem, but didn’t really find any solution. I then decided to boot the server back into unRAID, took one disk offline, converted it into an single Ext2 partition, and copied over all data from the other ReiserFS. After spending hours waiting for the copying to finish, I still had no luck mounting the newly created Ext2 partition under FreeBSD.
It then occurred to me that it could be an issue of the GPT partition table I had on the two hard disks, even though both disks were perfectly mountable and readable under Linux. I then used a tool called GPT fdisk to convert the partition table from GPT to MBR on the disk with the Ext2 partition, only to discover that it got worse: neither Linux nor FreeBSD would mount this disk any more!
Our of despair, I fired up fdisk under Linux, deleted and created a new MBR partition on the disk. Things magically worked from there, and no, the existing data on the Ext2 partition was not destroyed, so I didn’t have to copy again.
In retrospect, I couldn’t really recall what I did when I first attached these hard disks to the unRAID setup, perhaps I didn’t really setup GPT properly in the first place. So, yeah, partition tables are strange beasts, beware!
Then I got back to FreeBSD, after mounting the Ext2 partition from one disk (in my case, device ad6), I created ZFS on the other disk (in my case, device ad7) by the following command:
kldload zfs.ko
zpool create tank ad7The first line loads the ZFS kernel module. The next line turns an entire hard disk into a ZFS pool and name it “tank”. It’s also automatically mounted as /tank directory.
To ensure FreeBSD loads ZFS on its next boot, I did the following:
echo 'zfs_enable="YES"' >> /etc/rc.confI then copied everything from the Ext2 disk to ZFS. After the lengthy copying was done, the Ext2 disk was no longer needed. So I converted my existing ZFS to a mirror setup by attaching this freed up device:
zpool attach tank ad7 ad6where ad7 is my existing hard drive that had ZFS on it, and ad6 is the new drive I was adding to the pool. It became a mirror RAID setup. Pretty simple, isn’t it?
Well, maybe not. On the next reboot, I noticed that the system complained about a damaged GPT partition table on ad7. So I should have wiped the partition table on ad7 before I created ZFS on it. I ended up wasting a lot of time on this, but long story short, here is what I should have done:
/boot/zfs/zpool.cacheAnother reason that I went with a unRAID setup two years ago instead of a BSD variant such as FreeNAS was that I needed the machine to suspend and resume properly. ACPI was horrendously broken in FreeBSD back then. This time when I was contemplating the migration, I read that the support for ACPI was much improved since FreeBSD 8.0, so I decided to give it a shot.
And I was wrong.
Well, at least ACPI doesn’t fully work out-of-box on my hardware. I was able to suspend the system, but waking up was at best trial-and-error, testing my luck and patience. There were two problems:
The first wasn’t a big issue since I won’t be connecting a monitor to the server after it’s fully set up. But the second problem was a deal breaker, since my entire root partition was on a USB drive, and it would hang whenever I tried to invoke system commands. On the other hand, network devices continued to function properly on a resume, so that least that’s a good news.
After spending some time trying to debug and fix the problem but to no avail, I realized that in my original unRAID setup the entire root system was on a memdisk, so it didn’t have to resume a USB device at all. So now a possible way out was to boot FreeBSD from a USB memstick but off a root device completely in memory, not ulike what unRAID did.
I made a bootable USB memstick based on mfsBSD. The best way to do it is from within a FreeBSD system. I already had a working system (except for ACPI), so this was not a problem.
First, grab the latest ISO image from the mfsBSD site (in my case, 8.1-RELEASE-amd64), and mount it:
ftp http://mfsbsd.vx.sk/iso/mfsbsd-8.1-amd64.iso
setenv MDDEV `mdconfig -a -t vnode -o readonly -f mfsbsd-8.1-amd64.iso`
mkdir /mnt/iso
mount -r -t cd9660 /dev/${MDDEV} /mnt/isoNext, plug in the USB memstick (in my case, device da5), and partition, format, make it bootable, and mount it:
fdisk -BI /dev/da5
bsdlabel -wB da5s1
newfs /dev/da5s1a
boot0cfg -v -B da5
mkdir /mnt/usb
mount /dev/da5s1a /mnt/usbThen, we just copy all files from the ISO image to the memstick:
tar cf - -C /mnt/iso/ . | tar --unlink -pxf - -C /mnt/usb/Optionally, we can customize the mfsBSD loader. For example, let’s change the root password and the hostname, copy over the full set of kernel modules and pre-load a few of file system modules:
cp /boot/kernel/*.ko /mnt/usb/boot/kernel/
cat <<@END >> /mnt/usb/boot/loader.conf
mfsbsd.rootpw="foobar"
mfsbsd.hostname="tower"
nullfs_load="YES"
geom_label_load="YES"
ext2fs_load="YES"
msdosfs_load="YES"
ntfs_load="YES"
@ENDAlso optionally, we can customize the mfsroot memory image. For example, let’s turn on ntpdate, enable ZFS, set the time zone to PST, and generate a new set of SSH host keys:
gunzip /mnt/usb/mfsroot.gz
setenv MDDEV `mdconfig -a -t vnode -o readonly -f /mnt/usb/mfsroot`
mkdir /mnt/mfsroot /tmp/mfsroot
mount -r /dev/${MDDEV} /mnt/mfsroot
tar cf - -C /mnt/mfsroot/ . | tar --unlink -pxf - -C /tmp/mfsroot/
umount /mnt/mfsroot
mdconfig -d -u ${MDDEV}
cat <<@END >> /tmp/mfsroot/etc/rc.conf
ntpdate_enable="YES"
ntpdate_flags="north-america.pool.ntp.org"
zfs_enable="YES"
@END
cp /usr/share/zoneinfo/PST8PDT /tmp/mfsroot/etc/localtime
rm /tmp/mfsroot/etc/ssh/ssh_host*
ssh-keygen -t rsa1 -b 1024 -f /tmp/mfsroot/etc/ssh/ssh_host_key -N ''
ssh-keygen -t dsa -f /tmp/mfsroot/etc/ssh/ssh_host_dsa_key -N ''
ssh-keygen -t rsa -f /tmp/mfsroot/etc/ssh/ssh_host_rsa_key -N ''
makefs -f 10% /mnt/usb/mfsroot /tmp/mfsroot
gzip /mnt/usb/mfsrootWe can even add some new users, setup SSH authorized keys, etc, etc. Also, if we plan to have binary packages to load on boot-up, just put them under the /tmp/mfsroot/packages/ directory before the makefs step. One important thing to keep in mind is that this mfsroot memory image has to be less than 40MB, otherwise we are very like to hit a kernel panic on boot-up.
Now, we have the memstick ready, just umount and reboot:
umount /mnt/usb
rebootOn the next boot, be sure to select booting device to be the USB memdisk we just created, and it should boot successfully into mfsBSD, entirely memory based. We can log in with root user using the password we set in the loader.conf. Had we skipped that customization, the default password is mfsroot.
After I successfully booted into mfsBSD, I immediate verified that ACPI could now work properly, and ZFS (which uses the internal hard disk) still worked.
As a summary so far, I have:
I could have started with mfsBSD in the first place without going through a full FreeBSD installation on the portable HD. But then it would be difficult to prepare a memstick version of the mfsBSD ISO, unless I had another FreeBSD box handy. So even though there were some hiccups on the parition table, on ACPI, things are looking really good.
The mfsBSD distro is really quite minimal, and it is desirable to have a full BSD tree if we want to build and install packages through the FreeBSD port system. I opted to have a full base installation under ZFS.
This can be done in sysinstall in the following steps:
Install Root to be /bank/base/;Developer set;I did it differently by manually downloading all base packages from a BSD ftp site, but the above would work equally well. I didn’t choose to install ports since I wanted to do it separately (see below).
Now a full BSD tree is available under /bank/base/, but not everything is needed, and I wanted to have ZFS manage the mount point. So I did the following:
zfs create bank/db
zfs create bank/usr
zfs create bank/ports
zfs create bank/ports/distfiles
zfs set compression=on bank/ports
zfs set compression=off bank/ports/distfiles
tar cf - -C /bank/base/usr/ . | tar --unlink -pxf - -C /bank/usr/
rm -rf /bank/base/usr
zfs set mountpoint=/usr bank/usr
zfs set mountpoint=/usr/ports bank/ports
zfs set mountpoint=/usr/ports/distfiles bank/ports
zfs set mountpoint=/var/db bank/dbBasically I mounted bank/usr as /usr to override the existing memory disk mount in mfsBSD, and a compressed bank/ports (but with distrfiles uncompressed) under /usr/ports. I also wanted the /var/db to be persistent so I created a ZFS file system for it too.
Then we are ready to get the latest ports:
portsnap fetch
portsnap extractIt took quite a while, but when it’s done, I rebooted and verified everything was intact.
A final word on mfsBSD is that every system customization done to /etc/ must first be done to a live mfsroot tree before we produce a new mfsroot.gz and put it back to the USB memstick.
Previously I was using MediaTomb on the unRAID setup. It was kind of ok, but I decided to try something new this time. So I installed miniDLNA from the FreeBSD ports (net/minidlna, only available if you update to the latest port collection).
The configuration of miniDLNA was straightforward, and it took a little time to scan my media collections in the first run, but afterwards, it is ok. My TV, PS3 and Roku radio are all able to pick up things reasonably well. I like miniDLNA’s easy setup and small footprint compared to MediaTomb, but on the other hand, it has little support for transcoding while MediaTomb does.
One problem about video streaming though, is that devices usually have limited support for codecs, and especially when it comes to subtitles, neither my TV nor PS3 has any support for them. So transcoding is part of the reality. Unfortunately my server is only an Atom 230 box, not powerful enough to support real-time transcoding. So it doesn’t matter to me so much that miniDLNA does not support live transcoding, I have been doing it manually anyway. Though it would be really nice to have something like AirVideo, which adds new job to the transcoding queue (on the server side) directly from a device. Unfortunately, AirVideo only supports iPhone and iPad.
The most comprehensive guide I found online is x264 encoding guide, which uses mplayer to playback, x264 to encode video, mencoder to change framerate, nero to encode audio, and MP4Box to mux into the final mp4 container. Sounds complicated, doesn’t it?
Aside from giving the best result, it really takes a long time to finish all steps. Sometimes I just want to dump a video into an acceptable format for my TV and quick delete it after watching. Here is how I use mencoder to convert video.avi into video.mp4 (assuming we have a truetype font called ~/.fonts/msyh.ttf to render the subtitle file video.srt).
mencoder -spuaa 4 -spualign 2 -utf8 -subfont-text-scale 2.5 -nofontconfig \
-font ~/.fonts/msyh.ttf -of lavf -lavfopts format=mp4 -oac lavc -ovc lavc \
-lavcopts aglobal=1:vglobal=1:acodec=libfaac:vcodec=mpeg4:abitrate=128:vbitrate=1200:keyint=250:mbd=1:vqmax=10:lmax=10 \
-vf harddup video.avi -sub video.srt -o video.mp4Note that I set the subtitle font scale to be 2.5 (default is 5) because I have a 50" TV. You may want to stick to the default if you have a small screen.
If we have only VOB subtitles (two files: idx and sub), then we need to go through an extra step by embedding the subtitle into a MKV file before encoding with mencoder (which requires the mkvtoolnix package):
mkvmerge -o video.mkv video.avi video.idxWhen we use mencoder, just replace video.avi by video.mkv, and use -sid N to select subtitle track instead of using -sub video.srt.
As a side note, I was always amazed by the BSD port compilation, and this time it’s no exception. A mere make install under net/minidlna (and under shells/bash, since I did it first and cannot really tell their dependencies apart) got me a total of 84 packages!!!
> autoconf-2.68 libid3tag-0.15.1b
> autoconf-wrapper-20101119 libogg-1.2.2,4
> automake-1.11.1 libpthread-stubs-0.3_3
> automake-wrapper-20101119 libtheora-1.1.1_2
> bash-4.1.9 libtool-2.2.10
> bigreqsproto-1.1.0 libvorbis-1.3.2,3
> binutils-2.21 libvpx-0.9.5
> bison-2.4.3,1 libxcb-1.7
> damageproto-1.2.0 libxml2-2.7.8_1
> dri2proto-2.2 libxslt-1.1.26_2
> expat-2.0.1_1 m4-1.4.15,1
> faac-1.28_2 makedepend-1.0.2,1
> faad2-2.7_3,1 minidlna-2010.12.12
> ffmpeg-0.6.1_3,1 mp4v2-1.9.1
> fixesproto-4.1.1 mpfr-3.0.0
> flac-1.2.1_2 nasm-2.09.04,1
> gettext-0.18.1.1 openjpeg-1.3_2
> glproto-1.4.11 orc-0.4.11
> gmake-3.81_4 p5-Locale-gettext-1.05_3
> gmp-5.0.1 perl-5.10.1_3
> gpac-libgpac-0.4.5_4,1 pkg-config-0.25_1
> help2man-1.38.4 png-1.4.5
> inputproto-2.0 python26-2.6.6
> jbigkit-1.6 rsync-3.0.7
> jpeg-8_3 schroedinger-1.0.10
> kbproto-1.0.4 sqlite3-3.7.4
> lame-3.98.4 tcl-8.5.9
> libGL-7.4.4 tcl-modules-8.5.9
> libGLU-7.4.4 texi2html-1.82,1
> libX11-1.3.3_1,1 tiff-4.0.0
> libXau-1.0.5 unzip-6.0
> libXdamage-1.1.2 x264-0.110.1820
> libXdmcp-1.0.3 xcb-proto-1.6
> libXext-1.1.1,1 xcmiscproto-1.2.0
> libXfixes-4.0.4 xextproto-7.1.1
> libXxf86vm-1.1.0 xf86bigfontproto-1.2.0
> libcheck-0.9.8 xf86vidmodeproto-2.3
> libdrm-2.4.12_1 xorg-macros-1.6.0
> libexif-0.6.18_1 xproto-7.0.16
> libgcrypt-1.4.6 xtrans-1.2.5
> libgpg-error-1.10 xvid-1.2.2_1,1
> libiconv-1.13.1_1 yasm-1.1.0Quite a number of X11 libraries made into the list even though this is indented as a headless setup. Admittedly, some packages are needed only for compilation purposes (such as perl and python) but not for runtime. But still, it would really benefit everyone if package dependencies can be kept in control, especially for something like a miniDLNA server.
Or maybe it’s just my wishful thinking.
Since I’m quite new to ZFS, I’ll leave this section blank until I have something to report.
(TODO: my incremental backup script)