ZFS on FreeBSD is an incredible file system. It is fast, secure and flexible which makes it a pleasure to use. One of the current limitations when installing FreeBSD 9 is you can not install the OS to a ZFS root pool from inside the default installer. Using the guide from aisecure we were able to get the ZFS root install working by manually typing commands. What we really wanted was the script on the FreeBSD 9 install media. Our goal was to boot the install media, execute the script and have the script do the FULL FreeBSD install for us without user intervention. We are happy to say we have a solution.
On this page we are going to take a look at the script itself and then explain the simplest method we found to install a FreeBSD memstick image to a USB key and run our install script from the key. At that point you can complete a full install of FreeBSD on a ZFS root in around 2 minutes. Speed is neat, but the real advantage is you can do an install, play with the image for a while and re-install without worrying about wasting a lot of time reinstalling. Lets take a look at the script first.
You can copy and paste the following script if you like or download the same script from calomel.org : zfs.sh. The script is called "zfs.sh", but you can use any name you wish. Make sure to set the file to be executable or use "sh zfs.sh" to execute. We commented every section so take a look at the script before using it and make any changes you feel necessary for your environment. After this section we explain how to do the memstick install and how to use the fully operational memestick.
But first, what does the script do? The script will remove any partitions from the drive. This is necessary if you use the same install script on the same drive and are just testing installs. We want to make sure we have a clean partition every time. Then we make a new partition table and align the disk for 4K (4096 byte) sectors. The ZFS pool is created and options are set which we prefer like setting the checksum to fletcher4 and turning the access time bit off. You will notice we are asking ZFS to keep two(2) copies of every file. Additional copies and the use of fletcher4 checksums allow ZFS to automatically heal files if they are corrupted with a minimal performance hit. Then FreeBSD is installed from the memstick image. Finally, enable dhcp client to get the network up on boot, enable ssh and disable root logins and disable sendmail. You are welcome to add or delete options are you see fit. When the script is done you will have FreeBSD fully bootable from a ZFS pool on the target hard drive.
The script's default zroot size is "220G" for 220 gigabytes which easily fits on a 240 gigabyte SSD. Edit the script if you need a larger zroot. Use the label "G" for gigabytes and "T" for terabytes. Note: the script will error out if the partition is larger then the disk can support.
#!/bin/sh
set -euf
#
# Calomel.org
# https://calomel.org/zfs_freebsd_root_install.html
# FreeBSD 13.0-RELEASE ZFS Root Install script
# zfs.sh @ Version 0.26
# NOTE: ada0 for SATA , nvd0 for PCIe M.2 NVMe
echo "# remove any old partitions on destination drive"
umount zroot || true
umount /mnt || true
zpool destroy zroot || true
gpart delete -i 2 ada0 || true
gpart delete -i 1 ada0 || true
gpart destroy -F ada0 || true
echo ""
echo "# Create zfs boot (512k) and a 220 gig root partition"
gpart create -s gpt ada0
gpart add -a 4k -s 512k -t freebsd-boot ada0
gpart add -a 4k -s 220G -t freebsd-zfs -l disk0 ada0
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada0
echo ""
# Option 1: align to 4K, ashift=12
echo "# Align the Disks for 4K (ashift=12) and create the pool"
gnop create -S 4096 /dev/gpt/disk0
# Option 2: align to 8k, ashift=13
#echo "# Align the Disks for 8K (ashift=13) and create the pool"
#gnop create -S 8192 /dev/gpt/disk0
zpool create -f -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot /dev/gpt/disk0.nop
zpool export zroot
gnop destroy /dev/gpt/disk0.nop
zpool import -o altroot=/mnt -o cachefile=/var/tmp/zpool.cache zroot
echo ""
echo "# Set the bootfs property and set options"
zpool set bootfs=zroot zroot
zpool set listsnapshots=on zroot
zfs set logbias=throughput zroot
zfs set compression=lz4 zroot
zfs set atime=off zroot
zfs set copies=2 zroot
echo ""
echo "# Add swap space and apply options"
zfs create -V 2G zroot/swap
zfs set org.freebsd:swap=on zroot/swap
zfs set copies=1 zroot/swap
echo ""
echo "# Create a symlink to /home and fix some permissions"
cd /mnt/zroot ; ln -s usr/home home
echo ""
echo "# Set zfs to cache data for longer striped writes"
sysctl vfs.zfs.min_auto_ashift=12
sysctl vfs.zfs.trim.txg_batch=128
sysctl vfs.zfs.txg.timeout=900
sysctl vfs.zfs.vdev.def_queue_depth=128
sysctl vfs.zfs.vdev.write_gap_limit=0
echo ""
echo "# Install FreeBSD OS from *.txz memstick."
echo "# This will take a few minutes..."
cd /usr/freebsd-dist
export DESTDIR=/mnt/zroot
# Option 1: install a 64bit os, no 32bit libs or ports or source
for file in base.txz kernel.txz doc.txz;
# Option 2: only install a 64bit os, no 32bit libs
#for file in base.txz kernel.txz doc.txz ports.txz src.txz;
# Option 3: full freebsd install
#for file in base.txz lib32.txz kernel.txz doc.txz ports.txz src.txz;
do (cat $file | tar --unlink -xpJf - -C ${DESTDIR:-/}); done
echo ""
echo "# Copy zpool.cache to install disk."
cp /var/tmp/zpool.cache /mnt/zroot/boot/zfs/zpool.cache
echo ""
echo "# Setup ZFS root mount and boot"
echo 'zfs_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo 'zfs_load="YES"' >> /mnt/zroot/boot/loader.conf
echo 'vfs.root.mountfrom="zfs:zroot"' >> /mnt/zroot/boot/loader.conf
echo ""
echo "# use gpt ids instead of gptids or disks idents"
echo 'kern.geom.label.disk_ident.enable="0"' >> /mnt/zroot/boot/loader.conf
echo 'kern.geom.label.gpt.enable="1"' >> /mnt/zroot/boot/loader.conf
echo 'kern.geom.label.gptid.enable="0"' >> /mnt/zroot/boot/loader.conf
echo ""
echo "# enable networking, pf and ssh and stop syslog from listening."
echo 'hostname="FreeBSDzfs"' >> /mnt/zroot/etc/rc.conf
echo 'ifconfig_igb0="dhcp"' >> /mnt/zroot/etc/rc.conf
echo '#ifconfig_igb0="inet 192.168.0.150 netmask 255.255.255.0 ether 00:11:22:33:44:55"' >> /mnt/zroot/etc/rc.conf
echo '#defaultrouter="192.168.0.1"' >> /mnt/zroot/etc/rc.conf
echo '#pf_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo '#pflog_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo 'sshd_enable="YES"' >> /mnt/zroot/etc/rc.conf
echo 'syslogd_flags="-ss"' >> /mnt/zroot/etc/rc.conf
echo 'nameserver 1.1.1.1' >> /mnt/zroot/etc/resolv.conf
echo ""
echo "# drop packets sent to closed ports."
echo 'net.inet.icmp.drop_redirect=1' >> /mnt/zroot/etc/sysctl.conf
echo 'net.inet.sctp.blackhole=2' >> /mnt/zroot/etc/sysctl.conf
echo 'net.inet.tcp.blackhole=2' >> /mnt/zroot/etc/sysctl.conf
echo 'net.inet.tcp.drop_synfin=1' >> /mnt/zroot/etc/sysctl.conf
echo 'net.inet.tcp.path_mtu_discovery=0' >> /mnt/zroot/etc/sysctl.conf
echo 'net.inet.udp.blackhole=1' >> /mnt/zroot/etc/sysctl.conf
echo ""
echo "# sshd, disable remote root logins."
echo 'PermitRootLogin no' >> /mnt/zroot/etc/ssh/sshd_config
echo 'PermitEmptyPasswords no' >> /mnt/zroot/etc/ssh/sshd_config
echo ""
echo "# /etc/rc.conf disable sendmail"
echo 'dumpdev="NO"' >> /mnt/zroot/etc/rc.conf
echo 'sendmail_enable="NONE"' >> /mnt/zroot/etc/rc.conf
echo ""
echo "# touch the /etc/fstab else freebsd will not boot properly"
touch /mnt/zroot/etc/fstab
sync
echo ""
echo "# Syncing... Install Done."
echo ""
echo "# Hint: poweroff, remove the USB drive and re-boot the machine."
echo "# Then add a privlidged user to the 'wheel' group. You will"
echo "# then be able to ssh in as the new user and configure the box."
echo ""
sync
#### EOF ####
It takes a few steps to get a FreeBSD memstick image on the USB key and then finally put the script on it, but the task is not difficult. The positive side is once you make the key you can use it as many times as you want and save changes to the key as you modify configurations. This is an advantage a CD or DVD install does easily not offer. In fact, we keep a key chain of USB keys around of all different versions of FreeBSD for emergencies repairs and testing.
Step 1: Download FreeBSD memstick image. The zfs.sh script is version independent so you can get the latest stable 11 release or even 11 current. We always suggest getting the latest FreeBSD RELEASE image directly from freebsd.org (https).
curl -4JOL https://download.freebsd.org/ftp/releases/amd64/amd64/ISO-IMAGES/12.0/FreeBSD-12.0-RELEASE-amd64-memstick.img
Step 2: dd the memstick image to the USB key. Depending on the OS you are using depends on what the USB key device name is going to be. We are using a Xubuntu desktop and our main disk is /dev/sda . When we plug in the USB key it shows up in /var/log/messages as "sd 10:0:0:0: [sdb] Attached SCSI removable disk". This means the key can be referenced as /dev/sdb. To put the memstick image on to the USB key we are going to use the binary dd. The following shows our dd command putting FreeBSD 11 on the key. As dd is running you will not see any output, but you will see the light on the USB stick blinking. On our dd run a 732 MB image took 8 seconds at 91.2 MB/s to write to a SanDisk Extreme 128GB USB stick.
# FreeBSD 12.0 Release memstick image dd if=FreeBSD-12.0-RELEASE-amd64-memstick.img of=/dev/sdb bs=64k conv=sync
Step 3: Boot the USB key. When the image is done writing to the USB stick you can now boot off the USB device. On some motherboards you can tap F8 during the BIOS or UEFI post to bring up the boot menu where you can choose the device you want to boot from. When the USB key is chosen, FreeBSD will boot and you will be prompted with three options; Install, Shell or Live CD. For our purpose of downloading the zfs.sh script choose <Shell> to get to a privileged root shell.
Step 4: Mount root as Read/Write. In order to make changes to the USB key we need to remount the root file system on the key to read and write. Run the following mount command.
mount -rw /
Step 5: Enable networking. With the USB stick in read write mode it is time to enable networking for the purpose of getting the zfs.sh script on to the key. For this example we are choosing to use a dhcpd server on our network. First, move the current /etc/resolv.conf file, which is a symlink, out of the way. Then touch /etc/resolv.conf so dhclient can write to it. We want to enable dhclient to write out the DNS name servers the dhcpd network server responds with to the /etc/resolv.conf file. BTW, we have an Intel i350 network card so we use the igb0 interface name. You can use "ifconfig" to check the interface name of your network card.
rm /etc/resolv.conf; touch /etc/resolv.conf dhclient igb0
Step 6: Transfer zfs.sh script. The last task is to get the zfs.sh script onto the stick. You could choose any number of ways to copy the script using scp or ssh and cut and paste. We offer another option we personally use which is to download the script directly from calomel.org. Since you are on the network and have an ip, use fetch to collect the script and then chmod 755 to make the script executable.
fetch --no-verify-peer https://calomel.org/zfs.sh chmod 755 zfs.sh
Done! At this point you have a bootable USB key with the zfs.sh ZFS root install script. Now is a good time to look at the script and make any changes for your environment (vi zfs.sh). When you reboot using the key the file system will go back to read only mode.
Once you have the script on the memstick the install of FreeBSD to a hard disk or SSD is incredibly easy. These are the steps:
The script will run, destroy any previous partitions, make the new ZFS partition and install FreeBSD on the disk in a single ZFS pool. Once the install is done you can shut the machine down, remove the USB key and boot up the drive making sure it boots correctly. That's it, nice and simple.
Note: You can use any type of drive like an SSD, spinning hard drive or RAID of any size. It does not matter as the script will simply use all the space on the drive for ZFS.
ERROR ? If you get the error, "cannot mount '/mnt/zroot': failed to create mountpoint" make sure the USB drive is mounted read/write using the command, "mount -rw /".
To manually update FreeBSD we use a simple alias called "upgrayedd"; the name is a homage to Upgrayedd from the movie 'Idiocracy'. Add the alias to /root/.profile and execute as the root user on the command line. "upgrayedd" will download and update all packages and then look for system patches, downloaded the system patches if available and install the patches. The final task is remove any old packages in /var/db/freebsd-update/files/ because these old packages are not removed by the pkg tool. Notice the addition of "env PAGER=cat" to make sure "freebsd-update fetch" does not pause using the "more" command.
alias upgrayedd='/usr/sbin/pkg upgrade && echo "" && /usr/sbin/pkg autoremove && echo "" && /usr/sbin/pkg audit -F ; echo "" && env PAGER=cat freebsd-update fetch install; /usr/sbin/pkg clean -ay; for i in /var/db/freebsd-update/files/* ; do rm "$i"; done'
The following is the output of the "upgrayedd" alias when the local system is fully patched and up to date.
root@FreeBSDzfs: upgrayedd Updating FreeBSD repository catalogue... FreeBSD repository is up-to-date. All repositories are up-to-date. Checking for upgrades (1 candidates): 100% Processing candidates (1 candidates): 100% Checking integrity... done (0 conflicting) Your packages are up to date. Checking integrity... done (0 conflicting) Nothing to do. vulnxml file up-to-date 0 problem(s) in the installed packages found. src component not installed, skipped Looking up update.FreeBSD.org mirrors... 4 mirrors found. Fetching metadata signature for 12.0-RELEASE from update4.freebsd.org... done. Fetching metadata index... done. Fetching 1 metadata files... done. Inspecting system... done. Preparing to download files... done. No updates needed to update system to 12.0-RELEASE. No updates are available to install. Run '/usr/sbin/freebsd-update fetch' first. Nothing to do.
After setting up your new install using the ZFS root script from above you might want to add another disk of similar type and capacity to be a mirrored boot drive. When one of the boot drives die you will have all the boot drive data on the second drive and that second drive is able to boot on its own.
The first step is to install another disk in your machine. Since we installed the OS on the first disk on the SATA bus with the script above that drive will be known as "ada0". Use "dmesg | grep ada" to look at all recognized drives. When adding a second drive connected to the second SATA port this new drive will be "ada1".
The following commands will create two partitions on the new drive and attach the drive to the current zroot. ZFS will automatically create a mirror pool using both drives and the data from the original boot drive will be resilvered to the second drive. Make sure NOT to reboot the machine until the resilvering process is done. Check the progress using "zpool status".
The last step is to add the bootcode to the new mirror drive so when the first drive dies the second drive will be able to boot the machine. You can add the bootcode while the two drives are resilvering or wait until resilvering is done. When the resilver process completes the install is finished. At this point you will want to test for yourself to make sure the second drive boots when the first drive is unplugged. Once you are confident in the setup you are all done. We also have the same commands in a script called zfs_mirror.sh to be added the bootable usb stick to make spare disks.
root@FreeBSDzfs:~ # gpart create -s gpt ada1
root@FreeBSDzfs:~ # gpart add -a 4k -s 512k -t freebsd-boot ada1
root@FreeBSDzfs:~ # gpart add -a 4k -s 1T -t freebsd-zfs ada1
root@FreeBSDzfs:~ # zpool attach zroot ada0p2 ada1p2
Make sure to wait until resilver is done before rebooting.
If you boot from pool 'zroot', you may need to update
boot code on newly attached disk 'ada1'.
Assuming you use GPT partitioning and 'ada1' is your new boot disk
you may use the following command:
gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
root@FreeBSDzfs:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1
root@FreeBSDzfs:~ # zpool status
pool: zroot
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Tue Dec 25 10:20:30 2020
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: No known data errors
Once you have a mirrored ZFS boot pool, like we explained in the question above, you might decide to add a hot spare for added insurance against disk failure. If either one of the mirrored drives dies the hot spare will immediately be added to the mirrored pool and a copy of the data will be resilvered to the spare.
First, make sure your mirrored pool is healthy. We are going to add the third physical disk in the chassis which is ada2. ada2 is the same size as the other two mirrored disks. Create the exact same partition scheme as you used to make the mirrored boot drives and add the boot code to the spare drive. Finally add ada2 to the mirrored pool as a spare. The last "zpool status" verifies ada2 is now a spare for the zroot pool. You can add as many spares as you like.
root@FreeBSDzfs:~ # zpool status
pool: zroot
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Tue Dec 25 10:20:30 2020
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
errors: No known data errors
root@FreeBSDzfs:~ # gpart destroy -F ada2
root@FreeBSDzfs:~ # gpart create -s gpt ada2
root@FreeBSDzfs:~ # gpart add -a 4k -s 512k -t freebsd-boot ada2
root@FreeBSDzfs:~ # gpart add -a 4k -s 1T -t freebsd-zfs ada2
root@FreeBSDzfs:~ # gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada2
root@FreeBSDzfs:~ # zpool add zroot spare ada2
root@FreeBSDzfs:~ # zpool status
pool: zroot
state: ONLINE
scan: scrub repaired 0 in 0h0m with 0 errors on Tue Dec 25 10:20:30 2020
config:
NAME STATE READ WRITE CKSUM
zroot ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada0p2 ONLINE 0 0 0
ada1p2 ONLINE 0 0 0
spares
ada2 AVAIL
errors: No known data errors
Advanced Format is a term for any modern SSD or hard disk sector format storing data on disk which exceed 512 to 520 bytes per sector, such as the 4096-byte (4 KB) sectors. Larger 4k sectors use storage surface area more efficiently for large files but less efficiently for smaller files, and enable the integration of stronger error correction algorithms to maintain data integrity at higher storage densities.
In order for the drive to be properly aligned we need to verify both the partition is 4k aligned on the disk and that ZFS is configured to write to 4,096 byte sectors.
First, make sure gpart used the "-a 4k" directive to 4k align the partition like in the ZFS root install script above. Use "diskinfo" to query the disk partition and look for the "stripeoffset" value. Then take the modulus of the "stripeoffset" against 4096 bytes and, if the result is zero(0), the partition is 4k aligned on the disk. Here is an example of a drive after we used the ZFS root install script:
[root@zfsFBSD10 ~]# diskinfo -v ada0p2
ada0p2
512 # sectorsize
118111600640 # mediasize in bytes (110G)
230686720 # mediasize in sectors
0 # stripesize
----> 544768 # stripeoffset
228855 # Cylinders according to firmware.
16 # Heads according to firmware.
63 # Sectors according to firmware.
140879140000971400F7 # Disk ident.
[root@zfsFBSD10 ~]# echo 544768 % 4096 | bc
0
(if the result is zero(0) the partition is 4k aligned)
Second, check to make sure ZFS is writing the largest sector size of 4k. Using the "zdb" tool we look at the ZFS zroot pool and find the value for ashift. Make sure ashift equals twelve(12) meaning 4,096 byte sectors and NOT nine(9) which is only 512 byte sectors. The ashift value for 512 bytes is 9 (2^9 = 512) while the ashift value for 4,096 bytes is 12 (2^12 = 4,096). Note, ashift does NOT mean alignment, but only specifies that ZFS is configured to use 4,096 byte sectors. The following is the result of the same drive installed with the ZFS root script and you can see the value, "ashift: 12".
[root@zfsFBSD10 ~]# zdb
zroot:
version: 5000
name: 'zroot'
state: 0
txg: 11607
pool_guid: 10810981494469860258
hostid: 1232771827
hostname: 'calomel'
vdev_children: 1
vdev_tree:
type: 'root'
id: 0
guid: 10810981494469860258
children[0]:
type: 'disk'
id: 0
guid: 2636318580918461148
path: '/dev/ada0p2'
phys_path: '/dev/ada0p2'
whole_disk: 1
metaslab_array: 33
metaslab_shift: 30
----> ashift: 12
asize: 118106882048
is_log: 0
DTL: 52
create_txg: 4
features_for_read:
Yes. Compression reduces the amount of blocks written to the hard drive effectually increasing the aerial density of the disk. This means you can read and write data to the drive faster then with raw data. Some may say compression will slow down I/O access. This is not true as today's CPUs are wasting most of their time waiting for the hard drive to do their job. Check out our ZFS Raid Performance, Capacity and Integrity Comparison page and look for the section about compression.
Also, when installing FreeBSD using lz4 to a SSD boot disk we saw a savings of 2.22x compression. This reduces the amount of data written to the SSD and, some may argue, increases the lifetime of the SSD drive.
[root@zfsFBSD10 ~]# zfs get all zroot | grep compress zroot compressratio 2.22x - zroot compression lz4 local zroot refcompressratio 2.22x -
Indeed! Check out our ZFS Health Check and Status script. The script checks disk and volume errors as well as the health of the pool and even when the last zpool scrub was done.
How can I make a ZFS based USB drive for general data ?
A USB stick or USB drive is an excellent way to move data around the office or for backups. In fact we use a few usb sticks in important servers to do warm spare backups of configuration files. Those sticks are then rotated monthly to multiple off site, and more importantly, offline backup locations. At any one time we will have multiple warm and cold backups.
The following commands use gpart to create the 4K aligned partition and zpool to create a ZFS pool called "backup". When the USB key is mounted it will show up as "/backup" . The next set of zfs and zpool commands will set our preferred ZFS volume options on the USB stick including compression and keeping at least two(2) copies of each file for data consistency. You can never be too careful with your data.
#### Create a ZFS pool on a USB key (4k aligned) gpart destroy -F da0 gnop create -S 4096 da0 zpool create -f backup /dev/da0.nop zpool export backup gnop destroy /dev/da0.nop zpool import backup ### ZFS volume options zfs set atime=off backup zfs set compression=lz4 backup zfs set copies=2 backup zfs set logbias=throughput backup zpool set listsnapshots=on backup
After you have created the usb stick make sure you mount (import) and un-mount (export) the stick correctly to avoid data consistancy issues. You need to ensure the USB stick is ready to be removed by telling ZFS to write all data to the usb drive. Before removing the usb stick use "zpool export" and to mount the usb stick use "zpool import". The following are examples for the ZFS pool "backup" we made earlier.
## plug in the usb stick and mount with "import" zpool import backup ## Make sure to scrub the stick often to ensure data consistency. truthfully, ## we scrub every time we copy new data and before we unmount the usb stick zpool scrub backup ## To umount the USB stick use export BEFORE unplugging the USB key! zpool export backup
Any type of usb key can be used for ZFS, but understand there are some really slow usb sticks on the market. If you are purchasing a new USB drive we highly suggest a USB 3.0 key for your spare USB v3.0 motherboard slot. The reason is usb 2.0 is half duplex and limited to 35 MB/sec while usb 3.0 is full duplex and limited to 400 MB/sec. So, with usb 3.0 one can read and write at the same time and get full speeds from the usb stick while usb 2.0 can only communicate in one direction at a time.
Additionally, we recommend a USB 3.0 key which is engineered as a small SSD drive in back of a USB 3.0 interface. An excellent example of this design is the SanDisk Extreme 16 GB USB 3.0 Flash Drive SDCZ80-016G-X46 which you can get for around $25 US. We use many of these keys and easily sustain 50 MB/sec writes and 145 MB/sec reads formatted with our example ZFS pool and options on FreeBSD. In comparison, many generic usb 2.0 sticks average 8 MB/sec writes and 32 MB/sec reads.