Home           RSS           Search

April 18, 2014





FreeBSD ZFS Root Install Script

using the memstick image and a script to do a root ZFS install



ZFS on FreeBSD is an incredible file system. It is fast, secure and flexible which makes it a pleasure to use. One of the current limitations when installing FreeBSD 9 is you can not install the OS to a ZFS root pool from inside the default installer. Using the guide from aisecure we were able to get the ZFS root install working by manually typing commands. What we really wanted was the script on the FreeBSD 9 install media. Our goal was to boot the install media, execute the script and have the script do the FULL FreeBSD install for us without user intervention. We are happy to say we have a solution.

On this page we are going to take a look at the script itself and then explain the simplest method we found to install a FreeBSD memstick image to a USB key and run our install script from the key. At that point you can complete a full install of FreeBSD on a ZFS root in around 2 minutes. Speed is neat, but the real advantage is you can do an install, play with the image for a while and re-install without worrying about wasting a lot of time reinstalling. Lets take a look at the script first.




ZFS Root Install Script for FreeBSD 10.0 and 9.2

You can copy and paste the following script if you like or download the same script from calomel.org : zfs.sh. The script is called "zfs.sh", but you can use any name you wish. Make sure to set the file to be executable or use "sh zfs.sh" to execute. We commented every section so take a look at the script before using it and make any changes you feel necessary for your environment. After this section we explain how to do the memstick install and how to use the fully operational memestick.

But first, what does the script do? The script will remove any partitions from the drive. This is necessary if you use the same install script on the same drive and are just testing installs. We want to make sure we have a clean partition every time. Then we make a new partition table and align the disk for 4K (4096 byte) sectors. The ZFS pool is created and options are set which we prefer like setting the checksum to fletcher4 and turning the access time bit off. You will notice we are asking ZFS to keep three(3) copies of every file. Additional copies and the use of fletcher4 checksums allow ZFS to automatically heal files if they are corrupted with a very minimal performance hit. Then FreeBSD is installed from the memstick image. Finally, enable dhcp client to get the network up on boot, enable ssh and disable root logins and disable sendmail. You are welcome to add or delete options are you see fit. When the script is done you will have FreeBSD fully bootable from a ZFS pool on the target hard drive.

#!/bin/sh
#
# Calomel.org
#     https://calomel.org/zfs_freebsd_root_install.html
#     FreeBSD 10.0-RELEASE ZFS Root Install script
#     zfs.sh @ Version 0.19

echo "# remove any old partitions on destination drive"
umount zroot
umount /mnt
zpool destroy zroot
gpart delete -i 2 ada0
gpart delete -i 1 ada0
gpart destroy -F ada0

echo ""
echo "# Create zfs boot (512k) and zfs root partitions"
gpart create -s gpt ada0
gpart add -a 4k -s 512k -t freebsd-boot ada0
gpart add -a 4k -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0

echo ""
echo "# Align the Disks for 4K and create the pool"
gnop create -S 4096 /dev/gpt/disk0
zpool create -f -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot

echo ""
echo "# Set the bootfs property and set options"
zpool set bootfs=zroot zroot
zpool set listsnapshots=on zroot
#zpool set autoreplace=on zroot
#zpool set autoexpand=on zroot
zfs set checksum=fletcher4 zroot
zfs set compression=lz4 zroot
zfs set atime=off zroot
zfs set copies=3 zroot
#zfs set mountpoint=/ zroot

echo ""
echo "# Add swap space and apply options"
zfs create -V 1G zroot/swap
zfs set org.freebsd:swap=on zroot/swap

echo ""
echo "# Create a symlink to /home and fix some permissions"
cd /mnt/zroot ; ln -s usr/home home

echo ""
echo "# Install FreeBSD OS from *.txz memstick."
echo "# This will take a few minutes..."
cd /usr/freebsd-dist
export DESTDIR=/mnt/zroot

 # Option 1: only install a 64bit os without 32bit libs
  for file in base.txz kernel.txz doc.txz ports.txz src.txz;

 # Option 2: also include 32bit compatable libs
 #for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;

do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done

echo ""
echo "# Copy zpool.cache to install disk."
cp /var/tmp/zpool.cache /mnt/zroot/boot/zfs/zpool.cache

echo ""
echo "# Setup ZFS root mount and boot"
echo 'zfs_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/zroot/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/zroot/boot/loader.conf

echo ""
echo "# use gpt ids instead of gptids or disks idents"
echo 'kern.geom.label.disk_ident.enable="0"' >> /mnt/zroot/boot/loader.conf
echo 'kern.geom.label.gpt.enable="1"' >> /mnt/zroot/boot/loader.conf
echo 'kern.geom.label.gptid.enable="0"' >> /mnt/zroot/boot/loader.conf

echo ""
echo "# enable networking, pf and ssh and stop syslog from listening."
echo 'hostname="FreeBSDzfs"' >> /mnt/zroot/etc/rc.conf
echo 'ifconfig_igb0="dhcp"' >> /mnt/zroot/etc/rc.conf
echo '#ifconfig_igb0="inet 192.168.0.150 netmask 255.255.255.0 lladdr 00:11:22:33:44:55"' >> /mnt/zroot/etc/rc.conf
echo '#defaultrouter="192.168.0.1"' >> /mnt/zroot/etc/rc.conf
echo '#pf_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo '#pflog_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo 'sshd_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo 'syslogd_flags="-ss"' >> /mnt/zroot/etc/rc.conf

echo ""
echo "# sshd, disable remote root logins."
echo 'PermitRootLogin no' >> /mnt/zroot/etc/ssh/sshd_config
echo 'PermitEmptyPasswords no' >> /mnt/zroot/etc/ssh/sshd_config

echo ""
echo "# /etc/rc.conf disable sendmail"
echo 'dumpdev="NO"' >> /mnt/zroot/etc/rc.conf
echo 'sendmail_enable="NO"' >> /mnt/zroot/etc/rc.conf
echo 'sendmail_submit_enable="NO"' >> /mnt/zroot/etc/rc.conf
echo 'sendmail_outbound_enable="NO"' >> /mnt/zroot/etc/rc.conf
echo 'sendmail_msp_queue_enable="NO"' >> /mnt/zroot/etc/rc.conf

echo ""
echo "# touch the /etc/fstab else freebsd will not boot properly"
touch /mnt/zroot/etc/fstab

sync
echo ""
echo "# Syncing... Install Done."
echo ""
echo "# Hint: power down, remove the USB drive and re-boot the machine."
echo "#       Then add a privlidged user to the 'wheel' group. You will"
echo "#       then be able to ssh in as the new user and configure the box."
echo ""
sync

#### EOF ####




Making a USB memstick image with zfs.sh installed

It takes a few steps to get a FreeBSD memstick image on the USB key and then finally put the script on it, but the task is not difficult. The positive side is once you make the key you can use it as many times as you want and save changes to the key as you modify configurations. This is an advantage a CD or DVD install does easily not offer. In fact, we keep a key chain of USB keys around of all different versions of FreeBSD for emergencies repairs and testing.

Step 1: Download FreeBSD memstick image. The zfs.sh script is version independent so you can get the latest stable 9.2 release or even 10.0 current. We always suggest getting the latest FreeBSD 9.2 RELEASE image directly from freebsd.org (ftp14) or you can retrieve the FreeBSD 10.0 CURRENT memstick image from glenbarber.us. Special thanks to glenbarber for offering current builds of FreeBSD memstick for testing. It is really nice to use a clean install of 10.0 instead of installing 9 and upgrading.

Step 2: dd the memstick image to the USB key. Depending on the OS you are using depends on what the USB key device name is going to be. We are using a Xubuntu desktop and our main disk is /dev/sda . When we plug in the USB key it shows up in /var/log/messages as "sd 10:0:0:0: [sdb] Attached SCSI removable disk". This means the key can be referenced as /dev/sdb. To put the memstick image on to the USB key we are going to use the binary dd. The following shows our dd command putting either 9.2 or 10 on the key. As dd is running you will not see any output, but you will see the light on the USB stick blinking. On our dd run a 768 MB image took 78 seconds at 9.9 MB/s to write to the USB stick.

 # FreeBSD 9.2 Release memstick image
 dd if=FreeBSD-9.2-RELEASE-amd64-memstick.img of=/dev/sdb bs=64k conv=sync

-OR-

 ## FreeBSD 10.0 Current memstick image
 dd if=FreeBSD-10.0-RC1-amd64-memstick.img of=/dev/sdb bs=64k conv=sync

Step 3: Boot the USB key. When the image is done writing to the USB stick you can now boot off the USB device. On some motherboards you can tap F8 during the BIOS or UEFI post to bring up the boot menu where you can choose the device you want to boot from. When the USB key is chosen, FreeBSD will boot and you will be prompted with three options; Install, Shell or Live CD. For our purpose of downloading the zfs.sh script choose <Shell> to get to a privileged root shell.

Step 4: Mount root as Read/Write. In order to make changes to the USB key we need to remount the root file system on the key to read and write. Run the following mount command.

mount -rw /

Step 5: Enable networking. With the USB stick in read write mode it is time to enable networking for the purpose of getting the zfs.sh script on to the key. For this example we are choosing to use a dhcpd server on our network. First, move the current /etc/resolv.conf file, which is a symlink, out of the way. Then touch /etc/resolv.conf so dhclient can write to it. We want to enable dhclient to write out the DNS name servers the dhcpd network server responds with to the /etc/resolv.conf file. BTW, we have an Intel i350 network card so we use the igb0 interface name. You can use "ifconfig" to check the interface name of your network card.

rm /etc/resolv.conf; touch /etc/resolv.conf
dhclient igb0

Step 5: Transfer zfs.sh script. The last task is to get the zfs.sh script onto the stick. You could choose any number of ways to copy the script using scp or ssh and cut and paste. We offer another option we personally use which is to download the script directly from calomel.org. Since you are on the network and have an ip, use fetch to collect the script and then chmod 755 to make the script executable.

fetch --no-verify-peer https://calomel.org/zfs.sh
chmod 755 zfs.sh

Done! At this point you have a bootable USB key with the zfs.sh ZFS root install script. Now is a good time to look at the script and make any changes for your environment (vi zfs.sh). When you reboot using the key the file system will go back to read only mode.




How do I use the zfs.sh script from the memstick ?

Once you have the script on the memstick the install of FreeBSD to a hard disk or SSD is incredibly easy. These are the steps:

The script will run, destroy any previous partitions, make the new ZFS partition and install FreeBSD on the disk in a single ZFS pool. Once the install is done you can shut the machine down, remove the USB key and boot up the drive making sure it boots correctly. That's it, nice and simple.

Note: You can use any type of drive like an SSD, spinning hard drive or RAID of any size. It does not matter as the script will simply use all the space on the drive for ZFS.

ERROR ? If you get the error, "cannot mount '/mnt/zroot': failed to create mountpoint" make sure the USB drive is mounted read/write using the command, "mount -rw /".





Questions?


How can I add a second mirror boot disk ?

After setting up your new install using the ZFS root script from above you might want to add another disk of similar type and capacity to be a mirrored boot drive. When one of the boot drives die you will have all the boot drive data on the second drive and that second drive is able to boot on its own.

The first step is to install another disk in your machine. Since we installed the OS on the first disk on the SATA bus with the script above that drive will be known as "ada0". Use "dmesg | grep ada" to look at all recognized drives. When adding a second drive connected to the second SATA port this new drive will be "ada1".

The following commands will create two partitions on the new drive and attach the drive to the current zroot. ZFS will automatically create a mirror pool using both drives and the data from the original boot drive will be resilvered to the second drive. Make sure NOT to reboot the machine until the resilvering process is done. Check the progress using "zpool status".

The last step is to add the bootcode to the new mirror drive so when the first drive dies the second drive will be able to boot the machine. You can add the bootcode while the two drives are resilvering or wait until resilvering is done. When the resilver process completes the install is finished. At this point you will want to test for yourself to make sure the second drive boots when the first drive is unplugged. Once you are confident in the setup you are all done. We also have the same commands in a script called zfs_mirror.sh to be added the bootable usb stick to make spare disks.

root@FreeBSDzfs:~ # gpart create -s gpt ada1
root@FreeBSDzfs:~ # gpart add -a 4k -s 512k -t freebsd-boot ada1
root@FreeBSDzfs:~ # gpart add -a 4k -t freebsd-zfs ada1
root@FreeBSDzfs:~ # zpool attach zroot ada0p2 ada1p2
Make sure to wait until resilver is done before rebooting.

If you boot from pool 'zroot', you may need to update
boot code on newly attached disk 'ada1'.

Assuming you use GPT partitioning and 'ada1' is your new boot disk
you may use the following command:

        gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1

root@FreeBSDzfs:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
root@FreeBSDzfs:~ # zpool status
  pool: zroot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Tue Dec 25 10:20:30 2020
config:

        NAME           STATE     READ WRITE CKSUM
        zroot          ONLINE       0     0     0
          mirror-0     ONLINE       0     0     0
            ada0p2     ONLINE       0     0     0
            ada1p2     ONLINE       0     0     0

errors: No known data errors



How can I verify ZFS is 4k aligned on Advanced Format disks ?

Advanced Format is a term for any modern SSD or hard disk sector format storing data on disk which exceed 512 to 520 bytes per sector, such as the 4096-byte (4 KB) sectors. Larger 4k sectors use storage surface area more efficiently for large files but less efficiently for smaller files, and enable the integration of stronger error correction algorithms to maintain data integrity at higher storage densities.

In order for the drive to be properly aligned we need to verify both the partition is 4k aligned on the disk and that ZFS is configured to write to 4,096 byte sectors.

First, make sure gpart used the "-a 4k" directive to 4k align the partition like in the ZFS root install script above. Use "diskinfo" to query the partition and look for the "stripeoffset" value. Then take the modulus of the "stripeoffset" against 4096 bytes and, if the result is zero(0), the partition is 4k aligned on the disk. Here is an example of a drive after we used the ZFS root install script:

[root@zfsFBSD10 ~]# diskinfo -v ada0p2
ada0p2
        512             # sectorsize
        118111600640    # mediasize in bytes (110G)
        230686720       # mediasize in sectors
        0               # stripesize
        544768          # stripeoffset
        228855          # Cylinders according to firmware.
        16              # Heads according to firmware.
        63              # Sectors according to firmware.
        140879140000971400F7    # Disk ident.

[root@zfsFBSD10 ~]# echo 544768 % 4096 | bc
0

Second, check to make sure ZFS is writing the largest sector size of 4k. Using the "zdb" tool we look at the ZFS zroot pool and find the value for ashift. Make sure ashift equals twelve(12) meaning 4,096 byte sectors and NOT nine(9) which is only 512 byte sectors. The ashift value for 512 bytes is 9 (2^9 = 512) while the ashift value for 4,096 bytes is 12 (2^12 = 4,096). Note, ashift does NOT mean alignment, but only specifies that ZFS is configured to use 4,096 byte sectors. The following is the result of the same drive installed with the ZFS root script and you can see the value, "ashift: 12".

[root@zfsFBSD10 ~]# zdb 
zroot:
    version: 5000
    name: 'zroot'
    state: 0
    txg: 11607
    pool_guid: 10810981494469860258
    hostid: 1232771827
    hostname: 'calomel'
    vdev_children: 1
    vdev_tree:
        type: 'root'
        id: 0
        guid: 10810981494469860258
        children[0]:
            type: 'disk'
            id: 0
            guid: 2636318580918461148
            path: '/dev/ada0p2'
            phys_path: '/dev/ada0p2'
            whole_disk: 1
            metaslab_array: 33
            metaslab_shift: 30
---->       ashift: 12
            asize: 118106882048
            is_log: 0
            DTL: 52
            create_txg: 4
    features_for_read:



Does installing root with compression help performance ?

Yes. Compression reduces the amount of blocks written to the hard drive effectually increasing the aerial density of the disk. This means you can read and write data to the drive faster then with raw data. Some may say compression will slow down I/O access. This is not true as today's CPUs are wasting most of their time waiting for the hard drive to do their job. Check out our ZFS Raid Performance, Capacity and Integrity Comparison page and look for the section about compression.

Also, when installing FreeBSD using lz4 to a SSD boot disk we saw a savings of 2.22x compression. This reduces the amount of data written to the SSD and, some may argue, increases the lifetime of the SSD drive.

[root@zfsFBSD10 ~]# zfs get all zroot | grep compress
zroot  compressratio         2.22x                  -
zroot  compression           lz4                    local
zroot  refcompressratio      2.22x                  -



Do you have a ZFS Health Script we can use ?

Indeed! Check out our ZFS Health Check and Status script. The script checks disk and volume errors as well as the health of the pool and even when the last zpool scrub was done.



How can I make a ZFS based USB drive for general data ?

A USB stick or USB drive is an excellent way to move data around the office or for backups. In fact we use a few usb sticks in important servers to do warm spare backups of configuration files. Those sticks are then rotated monthly to multiple off site, and more importantly, offline backup locations. At any one time we will have multiple warm and cold backups.

The following commands use gpart to create the 4K aligned partition and zpool to create a ZFS pool called "backup". When the USB key is mounted it will show up as "/backup" . The next set of zfs and zpool commands will set our preferred ZFS volume options on the USB stick including compression and keeping at least three(3) copies of each file for data consistency. You can never be too careful with your data.

#### Create a ZFS pool on a USB key (4k aligned)
gpart destroy -F da0
gnop create -S 4096 da0
zpool create -f backup /dev/da0.nop
zpool export backup
gnop destroy /dev/da0.nop
zpool import backup

### ZFS volume options
zfs set compression=lz4 backup
zfs set copies=3 backup
zfs set atime=off backup
zfs set checksum=fletcher4 backup 
zpool set listsnapshots=on backup 
zpool set autoreplace=on backup
zpool set autoexpand=on backup

After you have created the usb stick make sure you mount (import) and un-mount (export) the stick correctly to avoid data consistancy issues. You need to ensure the USB stick is ready to be removed by telling ZFS to write all data to the usb drive. Before removing the usb stick use "zpool export" and to mount the usb stick use "zpool import". The following are examples for the ZFS pool "backup" we made earlier.

## plug in the usb stick and mount with "import"
zpool import backup

## Make sure to scrub the stick often to ensure data consistency. truthfully,
## we scrub every time we copy new data and before we unmount the usb stick
zpool scrub backup

## To umount the USB stick use export BEFORE unplugging the USB key!
zpool export backup

Any type of usb key can be used for ZFS, but understand there are some really slow usb sticks on the market. If you are purchasing a new USB drive we highly suggest a USB 3.0 key for your spare USB v3.0 motherboard slot. The reason is usb 2.0 is half duplex and limited to 35 MB/sec while usb 3.0 is full duplex and limited to 400 MB/sec. So, with usb 3.0 one can read and write at the same time and get full speeds from the usb stick while usb 2.0 can only communicate in one direction at a time.

Additionally, we recommend a USB 3.0 key which is engineered as a small SSD drive in back of a USB 3.0 interface. An excellent example of this design is the SanDisk Extreme 16 GB USB 3.0 Flash Drive SDCZ80-016G-X46 which you can get for around $25 US. We use many of these keys and easily sustain 50 MB/sec writes and 145 MB/sec reads formatted with our example ZFS pool and options on FreeBSD. In comparison, many generic usb 2.0 sticks average 8 MB/sec writes and 32 MB/sec reads.





Questions, comments, or suggestions? Contact Calomel.org or