Home           RSS           Search

March 20, 2014





FreeBSD Tuning and Optimization

performance modifications for 1gig and 10gig networks



The default install of FreeBSD and PC-BSD is quite fast and will work well the majority of the time. If you installed either FreeBSD or PC-BSD without any modifications you will not be disappointed. But, what if you wanted to get the most out of your install or you simply want to understand more about the OS ? In this post we offer some ideas to tune, tweak and optimize FreeBSD's network stack to get the most out of the operating system. Further down on the page we offer proofs to show gained performance and lower latency as well as links to the graphing tools we used so you can do the same.




FreeBSD is fast, but hardware is important

If you want to achieve optimized network throughput you need to use good hardware. As Monty Python taught us, it is daft to build a castle in a swamp. Cheap hardware will cause nothing but misery. High latency, low throughput and poor driver support not to mention general flakiness under load. A case in point is the built in network port on motherboards. The chipset may negotiate at one(1) gigabit, but it will not perform well under stress. Take some time, setup a good machine and you will be richly rewarded with "huge...tracks of land."

On our Network Tuning and Performance Guide we are using two hardware setups and these network modifications can be applied to both. The first is an example of a 1 gigabit machine for home or office use. The second is a rack mounted server for 10 gig and trunked 40 gigabit high speed networks. Both of these hardware configurations are actively used in production in clusters we support. We will post the hardware here, but again please take a look at our Network Tuning and Performance Guide for speed tests and many more details.

## Home or Office server (almost completely silent)

Processor    : AMD Athlon II X4 610e Propus 2.4GHz 45watt
CPU Cooler   : Zalman 9500A-LED 92mm 2 Ball CPU Cooler (fan off)
Motherboard  : Asus M4A89GTD Pro/USB3 AM3 AMD 890GX
Memory       : Kingston 4GB DDR3 KVR1333D3N9K2/4G
Hard Drive   : 256GB 2.5-inch SSD 840 Pro Series with ZFS root
               Western Digital Black WD4001FAEX 4TB 7200RPM SATA3/SATA 6.0 GB/s
Power Supply : Antec Green 380 watts EA-380D 
Case         : Antec LanBoy Air (completely fan-less)

Network Card : Intel PRO/1000 GT PCI PWLA8391GT PCI (two cards)
                  -OR-
               Intel I350-T2 Server Adapter (PCIe x4)

NOTE: The Intel I350-T2 incredibly fast and stable.
      The igb(4) driver is used for FreeBSD.

## Rack mounted server

Processor    : Intel Xeon L5630 Westmere 2.13GHz 12MB L3 Cache LGA 1366 40 Watt Quad-Core
Motherboard  : Supermicro X8ST3-F
Chassis      : SuperChassis 825TQ-R700LPV 2U rackmount (Revision K)
Memory       : KVR1333D3E9SK2/4G 4GB 1333MHz DDR3 ECC CL9 DIMM (Kit of 2) w/ Thermal Sensor
Hard Drive   : 256GB 2.5-inch SSD 840 Pro Series with ZFS root, LZ4 compression
Network Card : Myricom Myri-10G "Gen2" 10G-PCIE2-8B2-2S (PCI Express x8)
 Transceiver : Myricom Myri-10G SFP+ 10GBase-SR optical fiber transceiver (850nm wavelength)

Switches     : Arista 7150S-52    ##  Both switches were able to saturate 
             : Force10 S4810      ##  a bidirectional 10gig interface



The /boot/loader.conf

The /boot/loader.conf is where we setup the specifics for our network cards and some hashes. We tried to completely comment each of the options in the file. Directives which are commented out are not used and only included for reference. You should be able to copy and paste the following text into your loader.conf if you wish.

# FreeBSD 10.0 -- /boot/loader.conf  version 0.35
# https://calomel.org/freebsd_network_tuning.html
#
# low latency is important so we highly recommend that you disable hyper
# threading on Intel CPUs as it has an unpredictable affect on latency, cpu
# cache misses and load. 
#
# These settings are specifically tuned for a "low" latency FIOS (300/65) and
# gigabit LAN connections. If you have 10gig or 40gig you will need to increase
# the network buffers as proposed.
#

# ZFS root boot config
zfs_load="YES"
vfs.root.mountfrom="zfs:zroot"

# enable /dev/crypto for IPSEC of custom seeding using the AES-NI Intel
# hardware cpu support
#aesni_load="YES"

# load the Myri10GE kernel module on boot
#if_mxge_load="YES"

# load the PF CARP module
#if_carp_load="YES"

################################################ accf filters start ########
# accf accept filters are used so the server will not have to context switch several times
# before performing the initial parsing of the request. This could decrease server load by
# reducing the amount of CPU time to handle incoming requests.

# Wait for data accept filter. For nginx https servers add "listen
# 127.0.0.1:443 ssl spdy accept_filter=dataready;"
accf_data_load="YES"

# buffer incoming connections until complete HTTP requests arrive (nginx
# apache) for nginx http add, "listen 127.0.0.1:80 accept_filter=httpready;"
accf_http_load="YES"

# Wait for full DNS request accept filter (unbound)
accf_dns_load="YES"

################################################ accf filters end ##########

# Advanced Host Controller Interface (AHCI) 
ahci_load="YES"

# Asynchronous I/O, or non-blocking I/O is a form of input/output processing
# permitting other processing to continue before the transmission has finished.
aio_load="YES"

# reduce boot menu delay from 10 to 3 seconds. 
autoboot_delay="3"

# H-TCP Congestion Control for more aggressive increase in speed on higher
# latency, high bandwidth networks. 
cc_htcp_load="YES"

#thermal sensors for intel or amd cpus
#coretemp_load="YES"
amdtemp_load="YES"

################################################ intel igb start ############
# the following hw.igb.* are for the Intel i350 nic ( igb0, igb1, ect. )
# http://bsdrp.net/documentation/examples/forwarding_performance_lab_of_an_ibm_system_x3550_m3_with_intel_82580
# http://fasterdata.es.net/network-tuning/router-switch-buffer-size-issues/

# enable Intel's Adaptive Interrupt Moderation to reduce load for igb(4) (default 1)
#hw.igb.enable_aim="1"

# maximum number of interrupts per second generated by single igb(4) (default 8000)
#hw.igb.max_interrupt_rate="32000"

# number of queues supported on the hardware NIC (default 0),(Intel i350 = 8
# queues) For saturated network set to zero(0) which will auto tune and the
# driver will create as many queues as CPU cores up to a max of eight(8). For
# lightly loaded networks set to one(1) to reduce interrupts, lower latency and
# increase efficiency. (vmstat -i)
hw.igb.num_queues="1"

# enable MSI-X interrupts for PCI-E devices so nic polling is not needed anymore (default 1)
#hw.igb.enable_msix="1"

# igb under load will not drop packets by using transmit and receive descriptor
# rings in main memory which point to packet buffers. the igb driver transfers
# packet data to and from main memory independent of the CPU, using the
# descriptor rings as lists of packet transmit and receive requests to carry
# out. Increase each if your machine or network is saturated or if you have
# plenty of ram. (default 1024)
#hw.igb.txd="2048"
#hw.igb.rxd="2048"

# maximum number of received packets to process at a time, The default is too
# low for most firewalls. recommend "4096" or even "-1" which means unlimited
# (default 100)
hw.igb.rx_process_limit="4096"

################################################ intel igb end ##############

# hostcache cachelimit is the number of ip addresses in the hostcache list.
# Setting the value to zero(0) stops any ip address connection information from
# being cached and negates the need for "net.inet.tcp.hostcache.expire". We
# find disabling the hostcache increases burst data rates by 2x. A host cache
# is the client's cached tcp connection details and metrics (TTL, SSTRESH and
# VARTTL) the server can use to improve future performance of connections
# between the same two hosts. When a tcp connection is completed, our server
# will cache information about the connection until an expire timeout. If a new
# connection between the same client is initiated before the cache has expired,
# the connection will use the cached connection details to setup the
# connection's internal variables. This pre-cached setup allows the client and
# server to reach optimal performance significantly faster because the server
# will not need to go through the usual steps of re-learning the optimal
# parameters for the connection. To view the current host cache stats use
# "sysctl net.inet.tcp.hostcache.list" 
net.inet.tcp.hostcache.cachelimit="0"

# Change the pool to show the GPT id for each drive instead of the gptid or
# disk identifier. The gpt id will look like "ada0p2" with the first drive id
# "ada0" and partition string "p2". Use "gpart list" to see all drive
# identifiers and "zpool status" to see the chosen id through ZFS.
kern.geom.label.disk_ident.enable="0"
kern.geom.label.gpt.enable="1"
kern.geom.label.gptid.enable="0"

# when booting, display the ascii art FreeBSD Orb with the two horns on top.
# Just a cosmetic preference over "beastie", the multicolored daemon with
# pitchfork and big feet.
loader_logo="orb"

#
# #
#   #
################################################### OFF BELOW HERE #########
#
# Other options not used, but included for future reference. We found the
# following directives did not increase the speed or efficiency of our firewall
# over the defaults set by the developers. 

# higher HZ settings have a negative impact on machine performance due to
# handling more timer interrupts resulting in more context switches and cache
# flushes (default 1000).  Lower HZ settings can have a detrimental effect on
# ZFS.
# http://lists.freebsd.org/pipermail/freebsd-questions/2005-April/083482.html
# Also take a look into kern.sched.interact and kern.sched.slice in
# /etc/sysctl.conf
#kern.hz=1000

# increase the number of network mbufs the system is willing to allocate.  Each
# cluster represents approximately 2K of memory, so a value of 524288
# represents 1GB of kernel memory reserved for network buffers. (default
# 492680)
#kern.ipc.nmbclusters="492680"
#kern.ipc.nmbjumbop="246339"

# maximum number of interrupts per second on any interrupt level (vmstat -i for
# total rate). If you still see Interrupt Storm detected messages, increase the
# limit to a higher number and look for the culprit.  For 10gig NIC's set to
# 9000 and use large MTU. (default 1000)
#hw.intr_storm_threshold="9000"

# Size of the syncache hash table, must be a power of 2 (default 512)
#net.inet.tcp.syncache.hashsize="1024"

# Limit the number of entries permitted in each bucket of the hash table. (default 30)
#net.inet.tcp.syncache.bucketlimit="100"

# number of hash table buckets to handle incoming tcp connections. a value of
# 65536 allows the system to handle millions incoming connections. each tcp
# entry in the hash table on x86_64 uses 252 bytes of ram.  vmstat -z | egrep
# "ITEM|tcpcb" (default 65536 which is ~16M connections)
#net.inet.tcp.tcbhashsize="65536"

# For high bandwidth systems setting bindthreads to "0" will spread the
# network processing load over multiple cpus allowing the system to handle more
# throughput. The default is faster for most systems (default 0)
#net.isr.bindthreads="0"

# qlimit for igmp, arp, ether and ip6 queues only (netstat -Q) (default 256)
#net.isr.defaultqlimit="256"

# interrupt handling via multiple CPU (default direct)
#net.isr.dispatch="direct"

# limit per-workstream queues (use "netstat -Q" if Qdrop is greater then 0
# increase this directive) (default 10240)
#net.isr.maxqlimit="10240"

# Max number of threads for NIC IRQ balancing 3 for 4 cores in box leaving at
# least (default 1) one core for system or service processing. Again, if you
# notice one cpu being overloaded due to network processing this directive will
# spread out the load at the cost of cpu affinity unbinding. The default of "1"
# is faster.
#net.isr.maxthreads="1"

# network interface output queue length in number of packets. test your
# connection with the number of packets the interface can transmit in 50
# milliseconds. it is more efficient to send packets out of a queue then to
# re-send them from an application, especially from high latency wireless
# devices. warning: do not set to zero(0) or the network will not work due to
# "no network buffers" available. to set, if your upload speed is 25 megabit
# then set to around "107". Do NOT set too high as to avoid excessive buffer
# bloat. (default 50) calculate: bandwidth divided by 8 bits times 1000 divided
# by the MTU times 0.05 seconds ( ( (25/8) * 1000 ) / 1.460 ) * 0.05 = 107.02
# packets in 50 milliseconds.
#net.link.ifqmaxlen="107"

###########
# NOTE regarding "net.isr.*" : Processor affinity can effectively reduce cache
# problems but it does not curb the persistent load-balancing problem.[1]
# Processor affinity becomes more complicated in systems with non-uniform
# architectures. A system with two dual-core hyper-threaded CPUs presents a
# challenge to a scheduling algorithm. There is complete affinity between two
# virtual CPUs implemented on the same core via hyper-threading, partial
# affinity between two cores on the same physical chip (as the cores share
# some, but not all, cache), and no affinity between separate physical chips.
# It is possible that net.isr.bindthreads="0" and net.isr.maxthreads="3" can
# cause more slowdown if your system is not cpu loaded already. We highly
# recommend getting a more efficient network card instead of setting the
# "net.isr.*" options. Look at the Intel i350 for gigabit or the Myricom
# 10G-PCIE2-8C2-2S for 10gig. These cards will reduce the machines nic
# processing to 12% or lower.

# SIFTR (Statistical Information For TCP Research) is a kernel module which
# logs a range of statistics on active TCP connections to a log file in comma
# separated format. Only useful for researching tcp flows as it does add some
# processing load to the system.
# http://manpages.ubuntu.com/manpages/precise/man4/siftr.4freebsd.html
#siftr_load="YES"

# ZFS TUNING
#
# Increase VDEV read ahead cache size. This reduces scrub times and
# metadata-intensive tasks for a small cost in RAM. The vdev cache uses a
# simple FIFO rolling data set.
#vfs.zfs.vdev.cache.size="16M"
#
# Dynamically adjusted write limit based on previous txg commits to attempt
# to maintain a 3-second commit timeout
#vfs.zfs.txg.synctime_ms="3000"
# 
# Default vfs.zfs.write_limit_shift appears to be "3" which on a system
# with 2GB RAM such as this one results in a write_limit_max of 256MB, which
# is appropriate, so we're not going to change that here.
#vfs.zfs.write_limit_shift="2"

#
### EOF ###



The /etc/sysctl.conf

The /etc/sysctl.conf is the primary optimization file. Everything from congestion control to buffer changes can be found here. Again, each option we changed is fully commented and may also have a link to a research study for more information. Directives which are commented out are not used and included for reference. This is a large file so take some time to look through each option and understand why we made the change from default.

# FreeBSD 10.0 -- /etc/sysctl.conf  version 0.35
# https://calomel.org/freebsd_network_tuning.html
#
# low latency is important so we highly recommend that you disable hyper
# threading on Intel CPUs as it has an unpredictable affect on latency, cpu
# cache misses and load.
#
# These settings are specifically tuned for a "low" latency FIOS (300/65) and
# gigabit LAN connections. If you have 10gig or 40gig you will need to increase
# the network buffers as proposed.
#
# Before tuning the following two(2) sections on maxsockbuf and buf_max take 
# some time to read PSC's tips on Enabling High Performance Data Transfers.
# http://www.psc.edu/index.php/networking/641-tcp-tune

# Default is fine for most networks. You may want to increase to 4MB if the
# upload bandwidth is greater the 30Mbit. For 10GE hosts set to at least 16MB
# as well as to increase the TCP window size to 65535 and window scale to 9.
# For 10GE hosts with RTT over 100ms you will need to set a buffer of 150MB and
# a wscale of 12.  Default of "2097152 = 2*1024*1024" is fine for 1Gbit, FIOS
# or slower.
# network:   1 Gbit   maxsockbuf:    2MB   wsize:  6    2^6*65KB =    4MB (default)
# network:   1 Gbit   maxsockbuf:    4MB   wsize:  7    2^7*65KB =    8MB
# network:  10 Gbit   maxsockbuf:   16MB   wsize:  9    2^9*65KB =   32MB
# network:  40 Gbit   maxsockbuf:  150MB   wsize: 12   2^12*65KB =  260MB
# network: 100 Gbit   maxsockbuf:  600MB   wsize: 14   2^14*65KB = 1064MB
#kern.ipc.maxsockbuf=16777216  # (default 2097152)

# set auto tuning maximum to at least 16MB for 10GE hosts with RTT of less then
# 100ms. For 10GE hosts with RTT of greater then 100ms set buf_max to 150MB.
# The default of "2097152" is fine for 1Gb, FIOS or less. 
#net.inet.tcp.sendbuf_max=16777216  # (default 2097152)
#net.inet.tcp.recvbuf_max=16777216  # (default 2097152)

# H-TCP congestion control: algorithm which is more aggressive pushing up to max
# bandwidth (total BDP) and favors hosts with lower TTL / VARTTL then the
# default "newreno". Understand "newreno" works well in most conditions and
# enabling HTCP may only gain a you few percentage points of throughput. We
# suggest testing both, but we still prefer HTCP.
# http://www.sigcomm.org/sites/default/files/ccr/papers/2008/July/1384609-1384613.pdf
# make sure to also add 'cc_htcp_load="YES"' to /boot/loader.conf then check
# available congestion control options with "sysctl net.inet.tcp.cc.available"
net.inet.tcp.cc.algorithm=htcp  # (default newreno)

# H-TCP congestion control: adaptive backoff will increase bandwidth
# utilization by adjusting the additive-increase/multiplicative-decrease (AIMD)
# backoff parameter according to the amount of buffers available on the path.
# adaptive backoff ensures no queue along the path will remain completely empty
# after a packet loss event which increases buffer efficiency.
net.inet.tcp.cc.htcp.adaptive_backoff=1 # (default 0 ; disabled)

# H-TCP congestion control: RTT scaling will increase the fairness between
# competing TCP flows traversing different RTT paths through a common
# bottleneck. rtt_scaling increases the Congestion Window Size (CWND)
# independent of path round-trip time (RTT) leading to lower latency for
# interactive sessions when the connection is saturated by bulk data
# transfers. Default is 0 (disabled)
net.inet.tcp.cc.htcp.rtt_scaling=1 # (default 0 ; disabled)

# Ip Forwarding to allow packets to traverse between interfaces and is used for
# firewalls, bridges and routers. When fast IP forwarding is also enabled, IP packets
# are forwarded directly to the appropriate network interface with direct
# processing to completion, which greatly improves the throughput. All packets
# for local IP addresses, non-unicast, or with IP options are handled by the
# normal IP input processing path. All features of the normal (slow) IP
# forwarding path are supported by fast forwarding including firewall (through
# pfil(9) hooks) checking, except ipsec tunnel brokering. The IP fast
# forwarding path does not generate ICMP redirect or source quench messages
# though. Compared to normal IP forwarding, fastforwarding can give a speedup
# of 40 to 60% in packet forwarding performance which is great for interactive
# connections like online games or VOIP where low latency is critical.
net.inet.ip.forwarding=1      # (default 0)
net.inet.ip.fastforwarding=1  # (default 0)

# somaxconn is the OS buffer, backlog queue depth for accepting new TCP
# connections. Your application will have its own separate max queue length
# (maxqlen) which can be checked with "netstat -Lan". The default is 128
# connections per application thread. Lets say your Nginx web server normally
# receives 100 connections/sec and is single threaded application. If clients
# are bursting in at a total of 250 connections per/sec you may want to set the
# somaxconn at 512 to be a 512 deep connection buffer so the extra 122 clients
# (250-128=122) do not get denied service since you would have 412
# (512-100=412) extra queue slots. Also, a large listen queue will do a better
# job of avoiding Denial of Service (DoS) attacks if your application can
# handle the TCP load at the cost of more RAM and CPU time. Nginx sets is
# backlog queue to the same as the OS somaxconn by default.  Note:
# "kern.ipc.somaxconn" is not shown in "sysctl -a" output, but searching for
# "kern.ipc.soacceptqueue" gives the same value and both directives stand for 
# the same buffer value.
kern.ipc.soacceptqueue=1024  # (default 128 ; same as kern.ipc.somaxconn)

# maximum segment size (MSS) specifies the largest payload of data in a single TCP segment
# 1460 for a IPv4 1500 MTU network: MTU - 40 IPv4 header
# 8960 for a IPv4 9000 MTU network: MTU - 40 IPv4 header and switch ports set at 9216
# For most non-VPN networks an MTU of 1460 should work fine, but you may want
# to be cautious and use 1448. A smaller MSS allows an extra 12 bytes of space
# for additional TCP options like TCP time stamps (net.inet.tcp.rfc1323). If an
# MTU is too large, the packet must be fragmented by routers or the client
# which causes extra processing, delays and slow down of the transfer. If you
# are using PF with an outgoing scrub rule then PF will re-package the data
# using an MTU of 1460 by default, thus overriding this mssdflt setting.
# http://www.wand.net.nz/sites/default/files/mss_ict11.pdf 
net.inet.tcp.mssdflt=1460  # (default 536)

# Do not create a socket or compressed tcpw for TCP connections restricted to
# the local machine connecting to itself on localhost. An example connection
# would be a web server and a database server running on the same machine or
# freebsd jails connection to each other. 
net.inet.tcp.nolocaltimewait=1  # (default 0)

# SlowStart Flightsize is TCP's initial congestion window as the number of
# packets on the wire at the start of the connection or after congestion.
# Google recommends ten(10), so an MTU of 1460 bytes times ten(10) initial
# congestion window is a 14.6 kilobyte data burst which is considered small for
# today's internet connections. If you are running FreeBSD 9.1 or earlier we
# recommend a value of 10 or higher. Note, slowstart_flightsize was removed
# from FreeBSD 9.2 and now we can only set the initial congestion window to 10.
# http://www.igvita.com/2011/10/20/faster-web-vs-tcp-slow-start/ 
#net.inet.tcp.experimental.initcwnd10=1       # (default 1 for FreeBSD 10.0)
#net.inet.tcp.experimental.initcwnd10=1       # (default 0 for FreeBSD  9.2)
#net.inet.tcp.local_slowstart_flightsize=10   # (default 4 for FreeBSD  9.1)
#net.inet.tcp.slowstart_flightsize=10         # (default 4 for FreeBSD  9.1)

# The TCP window scale option is used to increase the TCP receive window size
# above its maximum value of 65,535 bytes (64k) as well as provide tcp time
# stamps allowing nearly every segment, including retransmissions, to be
# accurately timed at negligible computational cost.
#net.inet.tcp.rfc1323=1  # (default 1)

# Make sure rfc3390 is disabled in FreeBSD 9.1 so the slowstart flightsize
# values above are used. In FreeBSD 9.2 the experimental.initcwnd10 has a
# higher priority then rfc3390 so we can keep rfc3390 on.
# http://lists.freebsd.org/pipermail/svn-src-head/2012-October/041733.html
#net.inet.tcp.rfc3390=1  # (default 1)

# When running a web server, we suggest increasing the sendspace to the total
# size of an average web page you are serving. For example, if a user requests
# your home page which contains two(2) pictures, css and index.html equaling
# 212 kilobytes then set the sendspace to something like 262144 (256K). This
# will allow the web server to quickly dump the entire requested page set into
# the network buffer (SOCK_STREAM) and free web server resources. Even though
# Nginx is a non-blocking processes we saved 200 milliseconds sending data to
# clients. You may also want to test decreasing the recvspace to better resist
# DoS attacks or set your webserver to a similar sized receive buffer, for
# Nginx set "listen 80 default rcvbuf=16k;" for a "small" sixteen(16)
# kilobyte receive buffer.
#net.inet.tcp.sendspace=262144  # (default 32768)
#net.inet.tcp.recvspace=262144  # (default 65536)

# Increase auto-tuning TCP step size of the TCP transmit and receive buffers
# for multi-gigabit networks. The buffer starts at "net.inet.tcp.sendspace"
# and "net.inet.tcp.recvspace" and increases by these increments up to
# "net.inet.tcp.recvbuf_max" and "net.inet.tcp.sendbuf_max" as auto tuned by
# FreeBSD.  http://fasterdata.es.net/host-tuning/freebsd/ 
#net.inet.tcp.sendbuf_inc=262144  # (default 8192 )
#net.inet.tcp.recvbuf_inc=262144  # (default 16384)

# Reduce the amount of SYN/ACKs we will re-transmit to an unresponsive client.
# On a client's initial connection our server will always send a SYN/ACK in
# response to the client's initial SYN. Limiting retranstited SYN/ACKS reduces
# local syn cache size and a "SYN flood" DoS attack's collateral damage by not
# sending SYN/ACKs back to spoofed ips multiple time. If we do continue to send
# SYN/ACKs to spoofed IPs they may send RST back to us and an "amplification"
# attack would begin against our host. If you do not wish to send retransmits
# at all then set to zero(0) especially if you are under a SYN attack. Also set
# "net.inet.tcp.msl" to two(2) times the average round trip time of a client,
# but no lower then 5000ms (5s).
# http://www.ouah.org/spank.txt
# http://people.freebsd.org/~jlemon/papers/syncache.pdf
net.inet.tcp.syncache.rexmtlimit=1  # (default 3)

# Spoofed packet attacks may be used to overload the kernel route cache. A
# spoofed packet attack uses random source IPs to cause the kernel to generate
# a temporary cached route in the route table, Route cache is an extraneous
# caching layer mapping interfaces to routes to IPs and saves a lookup to the
# Forward Information Base (FIB); a routing table within the network stack. The
# IPv4 routing cache was intended to eliminate a FIB lookup and increase
# performance. While a good idea in principle, unfortunately it provided a very
# small performance boost in less than 10% of connections and opens up the
# possibility of a DoS vector. Setting rtexpire and rtminexpire to two(2)
# seconds should be sufficient to protect the route table from attack.
# http://www.es.freebsd.org/doc/handbook/securing-freebsd.html
net.inet.ip.rtexpire=2       # (default 3600)
net.inet.ip.rtminexpire=2    # (default 10  )
#net.inet.ip.rtmaxcache=128  # (default 128 )

# Syncookies have a certain number of advantages and disadvantages. Syncookies
# are useful if you are being DoS attacked as this method helps filter the
# proper clients from the attack machines. But, since the TCP options from the
# initial SYN are not saved in syncookies, the tcp options are not applied to
# the connection, precluding use of features like window scale, timestamps, or
# exact MSS sizing. As the returning ACK establishes the connection, it may be
# possible for an attacker to ACK flood a machine in an attempt to create a
# connection. Another benefit to overflowing to the point of getting a valid
# SYN cookie is the attacker can include data payload. Now that the attacker
# can send data to a FreeBSD network daemon, even using a spoofed source IP
# address, they can have FreeBSD do processing on the data which is not
# something the attacker could do without having SYN cookies. Even though
# syncookies are helpful during a DoS, we are going to disable them at this
# time.
net.inet.tcp.syncookies=0  # (default 1)

# Intel igb0 network card driver tuning
# http://www.intel.com/content/dam/doc/application-note/gbe-controllers-interrupt-moderation-appl-note.pdf
#dev.igb.0.fc=0                       # disable flow control for intel nics. isp's may abuse flow control to slow down
                                      # customers even though you are not using your full bandwidth. (default 3)

# General Security and DoS mitigation.
#net.bpf.optimize_writers=0           # bpf are write-only unless program explicitly specifies the read filter (default 0)
#net.bpf.zerocopy_enable=0            # zero-copy BPF buffers, breaks dhcpd ! (default 0)
net.inet.ip.check_interface=1         # verify packet arrives on correct interface (default 0)
#net.inet.ip.portrange.randomized=1   # randomize outgoing upper ports (default 1)
net.inet.ip.process_options=0         # IP options in the incoming packets will be ignored (default 1)
#net.inet.ip.random_id=1              # assign a random IP_ID to each packet leaving the system (default 0)
net.inet.ip.redirect=0                # do not send IP redirects (default 1)
#net.inet.ip.accept_sourceroute=0     # drop source routed packets since they can not be trusted (default 0)
#net.inet.ip.sourceroute=0            # if source routed packets are accepted the route data is ignored (default 0)
net.inet.ip.stealth=1                 # do not reduce the TTL by one(1) when a packets goes through the firewall (default 0)
#net.inet.icmp.bmcastecho=0           # do not respond to ICMP packets sent to IP broadcast addresses (default 0)
#net.inet.icmp.maskfake=0             # do not fake reply to ICMP Address Mask Request packets (default 0)
#net.inet.icmp.maskrepl=0             # replies are not sent for ICMP address mask requests (default 0)
#net.inet.icmp.log_redirect=0         # do not log redirected ICMP packet attempts (default 0)
net.inet.icmp.drop_redirect=1         # no redirected ICMP packets (default 0)
#net.inet.icmp.icmplim=500            # number of ICMP/TCP RST packets/sec, increase for bittorrent or many clients. (default 200)
#net.inet.icmp.icmplim_output=1       # show "Limiting open port RST response" messages (default 1)
#net.inet.tcp.always_keepalive=0      # tcp keep alive detection for dead peers, can be spoofed (default 1)
net.inet.tcp.drop_synfin=1            # SYN/FIN packets get dropped on initial connection (default 0)
#net.inet.tcp.ecn.enable=1            # explicit congestion notification (ecn) warning: some ISP routers abuse ECN (default 0)
net.inet.tcp.fast_finwait2_recycle=1  # recycle FIN/WAIT states quickly (helps against DoS, but may cause false RST) (default 0)
net.inet.tcp.icmp_may_rst=0           # icmp may not send RST to avoid spoofed icmp/udp floods (default 1)
#net.inet.tcp.maxtcptw=15000          # max number of tcp time_wait states for closing connections (default 5120)
net.inet.tcp.msl=5000                 # 5s maximum segment life waiting for an ACK in reply to a SYN-ACK or FIN-ACK (default 30000)
net.inet.tcp.path_mtu_discovery=0     # disable MTU discovery since most ICMP type 3 packets are dropped by others (default 1)
#net.inet.tcp.rfc3042=1               # limited transmit mechanism which can smooth out burst lose transmissions (default 1)
#net.inet.tcp.sack.enable=1           # TCP Selective Acknowledgments are needed for high throughput (default 1)
net.inet.udp.blackhole=1              # drop udp packets destined for closed sockets (default 0)
net.inet.tcp.blackhole=2              # drop tcp packets destined for closed ports (default 0)
#net.route.netisr_maxqlen=2048        # route queue length (rtsock using "netstat -Q") (default 256)
security.bsd.see_other_uids=0         # only allow users to see their own processes. root can see all (default 1)



######### OFF BELOW HERE #########
#
# Other options not enabled, but included for future reference. The following
# may be needed in high load environments or against DDOS attacks. Take a look
# at the detailed comments for more information and make an informed decision.

# host cache is the client's cached tcp connection details and metrics (TTL,
# SSTRESH and VARTTL) the server can use to improve future performance of
# connections between the same two hosts. When a tcp connection is completed,
# our server will cache information about the connection until an expire
# timeout. If a new connection between the same client is initiated before the
# cache has expired, the connection will use the cached connection details to
# setup the connection's internal variables. This pre-cached setup allows the
# client and server to reach optimal performance significantly faster because
# the server will not need to go through the usual steps of re-learning the
# optimal parameters for the connection. Unfortunately, this can also make
# performance worse because the hostcache will apply the exception case to
# every new connection from a client within the expire time. In other words, in
# some cases, one person surfing your site from a mobile phone who has some
# random packet loss can reduce your server's performance to this visitor even
# when their temporary loss has cleared.  3900 seconds allows clients who
# connect regularly to stay in our hostcache. To view the current host cache
# stats use "sysctl net.inet.tcp.hostcache.list" . If you have
# "net.inet.tcp.hostcache.cachelimit=0" like in our /boot/loader.conf example
# then this expire time is negated and not used.
#net.inet.tcp.hostcache.expire=3900  # (default 3600)

# By default, acks are delayed by 100 ms or sent every other packet in order to
# improve the chance of being added to another returned data packet which is
# full. This method can cut the number of tiny packets flowing across the
# network and is efficient. But, delayed ACKs cause issues on modern, low
# latency networks. TCP works by increasing the congestion window, which is the
# amount of data currently traveling on the wire, based on the number of ACKs
# received per time frame. Delaying the number of ACKs received results in less
# data on the wire, time in slow start is doubled and in congestion avoidance
# after packet loss the congestion window growth is slowed. If you are sending
# time sensitive, burst data like interactive web traffic try disabling delayed
# acks. Make sure "net.inet.tcp.sack.enable" and "net.inet.tcp.rfc1323" are
# also enabled. Setting delacktime higher then 100 will to slow down downloads
# as ACKs are queued too long.
# http://www.tel.uva.es/personales/ignmig/pdfs/ogonzalez_NOC05.pdf
#net.inet.tcp.delayed_ack=0   # (default 1)
#net.inet.tcp.delacktime=30  # (default 100)

# security settings for jailed environments. it is generally a good idea to
# separately jail any service which is accessible by an external client like
# the web or mail server. This is especially true for public facing services.
# take a look at ezjail, http://forums.freebsd.org/showthread.php?t=16860
#security.jail.allow_raw_sockets=1       # (default 0)
#security.jail.enforce_statfs=2          # (default 2)
#security.jail.set_hostname_allowed=0    # (default 1)
#security.jail.socket_unixiproute_only=1 # (default 1)
#security.jail.sysvipc_allowed=0         # (default 0)
#security.jail.chflags_allowed=0         # (default 0)

# decrease the scheduler maximum time slice for lower latency program calls.
# by default we use stathz/10 which equals thirteen(13). also, decrease the
# scheduler maximum time for interactive programs as this is a dedicated
# server (default 30). Also make sure you look into "kern.hz=100" in /boot/loader.conf
#kern.sched.interact=5 # (default 30)
#kern.sched.slice=3    # (default 12)

# increase localhost network buffers. For example, if you run many high
# bandwidth services on lo0 like an http or local DB server and forward public
# external traffic using Pf. Also, if running many jails on lo0 then these may
# help. set to 10x(lo0 mtu 16384 + 40 bytes for header) = 164240
#net.local.stream.sendspace=164240  # (default 8192)
#net.local.stream.recvspace=164240  # (default 8192)

# ZFS L2ARC tuning - If you have read intensive workloads make sure to use an
# SSD for your L2ARC. Verify noprefetch is enabled(1) and increase the speed at
# which the system can fill the L2ARC device. By default, when the L2ARC is
# being populated FreeBSD will only write at 16MB/sec to the SSD. 16MB
# calculated by adding the speed of write_boost and write_max. 16MB/sec is way
# too slow as many SSD's made today which can easily sustain 500MB/sec. We
# highly recommend setting both write_boost and write_max to at least 256MB each
# so the L2ARC can be seeded quickly. Contrary to myth, enterprise class SSDs
# can last for many years under constant read/write abuse of a web server.
#vfs.zfs.l2arc_noprefetch=1          # (default 1)
#vfs.zfs.l2arc_write_boost=268435456 # (default 8388608)
#vfs.zfs.l2arc_write_max=268435456   # (default 8388608)

# ZFS - Set TXG write limit to a lower threshold. This helps "level out" the
# throughput rate (see "zpool iostat").  A value of 256MB works well for
# systems with 4 GB of RAM, while 1 GB works well for us w/ 8 GB on disks which
# have 64 MB cache.
#vfs.zfs.write_limit_override=1073741824
 
# This is a pure file server so commit async writes after 60 seconds if
# vfs.zfs.write_limit_max has not been reached by this timeout. Setting the
# timeout higher then the default 5 seconds leads to high load, bursting
# writes. If a user is on the system the machine system to be unresponsive.
# Test this before using.
#vfs.zfs.txg.timeout=60
 
# For slow drives, set outstanding vdev I/O to "1" to prevent parallel
# reads/writes per zfs vdev. By limiting read write streams we effectually force
# drive access into long sequential disk access for drives like a single
# 5400rpm disk. A value of one is not good for multiple disk spindles.
#vfs.zfs.vdev.min_pending="1"
#vfs.zfs.vdev.max_pending="1"

# TCP keep alive can help detecting network errors and signaling connection
# problems. Keep alives will increase signaling bandwidth used, but as
# bandwidth utilized by signaling channels is low from its nature, the increase
# is insignificant. the system will disconnect a dead TCP connection when the
# remote peer is dead or unresponsive for: 10000 + (5000 x 8) = 50000 msec (50
# sec)
#net.inet.tcp.keepidle=10000     # (default 7200000 )
#net.inet.tcp.keepintvl=5000     # (default 75000 )
#net.inet.tcp.always_keepalive=1 # (default 1)

# maximum incoming and outgoing ip4 network queue sizes. if, and only if,
# "sysctl net.inet.ip.intr_queue_drops" is greater
# then zero increase these values till queue_drops is always zero(0).
#net.inet.ip.intr_queue_maxlen=4096
#net.route.netisr_maxqlen=4096

# UFS hard drive read ahead equivalent to 4 MiB at 32KiB block size. Easily
# increases read speeds from 60 MB/sec to 80 MB/sec on a single spinning hard
# drive. Samsung 830 SSD drives went from 310 MB/sec to 372 MB/sec (SATA 6).
# use Bonnie++ to performance test file system I/O
#vfs.read_max=128

# global limit for number of sockets in the system. If kern.ipc.numopensockets
# plus net.inet.tcp.maxtcptw is close to kern.ipc.maxsockets then increase this
# value
#kern.ipc.maxsockets = 25600

# spread tcp timer callout load evenly across cpus. We did not see any speed
# benefit from enabling per cpu timers. The default is off(0)
#net.inet.tcp.per_cpu_timers = 0

# Increase maxdgram length for jumbo frames (9000 mtu) OSPF routing. Safe for
# 1500 mtu too.
#net.inet.raw.maxdgram=9216
#net.inet.raw.recvspace=9216 

# seeding cryptographic random number generators is provided by the /dev/random
# device, which provides psudo "real" randomness. The arc4random(3) library call
# provides a pseudo-random sequence which is generally reckoned to be suitable
# for simple cryptographic use. The OpenSSL library also provides functions for
# managing randomness via functions such as RAND_bytes(3) and RAND_add(3). Note
# that OpenSSL uses the random device /dev/random for seeding automatically.
# http://manpages.ubuntu.com/manpages/lucid/man4/random.4freebsd.html
#kern.random.yarrow.gengateinterval=10  # default 10 [4..64]
#kern.random.yarrow.bins=10             # default 10 [2..16]
#kern.random.yarrow.fastthresh=192      # default 192 [64..256]
#kern.random.yarrow.slowthresh=256      # default 256 [64..256]
#kern.random.yarrow.slowoverthresh=2    # default 2 [1..5]
#kern.random.sys.seeded=1               # default 1
#kern.random.sys.harvest.ethernet=1     # default 1
#kern.random.sys.harvest.point_to_point=1 # default 1
#kern.random.sys.harvest.interrupt=1    # default 1
#kern.random.sys.harvest.swi=0          # default 0 and actually does nothing when enabled

# IPv6 Security
# For more info see http://www.fosslc.org/drupal/content/security-implications-ipv6
# Disable Node info replies
# To see this vulnerability in action run `ping6 -a sglAac ::1` or `ping6 -w ::1` on unprotected node
#net.inet6.icmp6.nodeinfo=0
# Turn on IPv6 privacy extensions
# For more info see proposal http://unix.derkeiler.com/Mailing-Lists/FreeBSD/net/2008-06/msg00103.html
#net.inet6.ip6.use_tempaddr=1
#net.inet6.ip6.prefer_tempaddr=1
# Disable ICMP redirect
#net.inet6.icmp6.rediraccept=0
# Disable acceptation of RA and auto linklocal generation if you don't use them
##net.inet6.ip6.accept_rtadv=0
##net.inet6.ip6.auto_linklocal=0

#
### EOF ###



OPTIONAL: Rebuilding the Kernel

Rebuilding the FreeBSD kernel is completely optional and will not affect the speed test we show. We simply wanted to include the modifications we made to the test machine kernel for completeness. This kernel build turns out to be around 59 megabytes in size compared to the default 376 megabytes.

We have removed all sound card, wireless and other drivers because our server did not need them. The only network card card driver is for the Intel igb and em. Your setup might have a different NIC so compare this configuration with the defaults and make an informed decision.

# FreeBSD 10.0 -- /etc/sysctl.conf  version 0.35
# https://calomel.org/freebsd_network_tuning.html
#
#############################################

## Rebuild the kernel with the following
# kldstat
# echo "MODULES_OVERRIDE=zfs opensolaris aio accf_data accf_dns accf_http amdtemp cc/cc_htcp fdescfs geom/geom_nop usb/ums" >> /etc/make.conf
# cp /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/CALOMEL
# vi /usr/src/sys/amd64/conf/CALOMEL
# cd /usr/src/ && make clean && make buildkernel KERNCONF=CALOMEL && make installkernel KERNCONF=CALOMEL && make clean && sync && reboot

ident CALOMEL
cpu   HAMMER

# change kernel message color to green on black
# while all other text is white on black
options SC_KERNEL_CONS_ATTR=(FG_GREEN|BG_BLACK)

# enable Pf without ALTQ (HFSC)
device pf
device pflog
device pfsync
options ALTQ
options ALTQ_HFSC
options ALTQ_NOPCC

# forward packets without decrementing
# the time to live (TTL) counter
options IPSTEALTH

# IPsec (Internet Protocol Security)
#options IPSEC
#options IPSEC_NAT_T
#device  crypto

# Crypto and AES-NI support
# also add aesni_load="YES" to /boot/loader.conf
# verify AES speeds before and after with "openssl speed -evp aes-256-cbc -elapsed"
# as /dev/crypto can be slower then no /dev/crypto on some systems. Also, the
# ports version of /usr/local/bin/openssl is significantly faster then the
# default /usr/bin/openssl version which comes with freebsd.
#device crypto
#device cryptodev
#device aesni

#############################################

makeoptions DEBUG=-g        # Build kernel with gdb(1) debug symbols
makeoptions WITH_CTF=1      # Run ctfconvert(1) for DTrace support

options     SCHED_ULE       # ULE scheduler
options     PREEMPTION      # Enable kernel thread preemption
options     INET            # InterNETworking
options     INET6           # IPv6 communications protocols
options     TCP_OFFLOAD     # TCP offload
options     SCTP            # Stream Control Transmission Protocol
options     FFS             # Berkeley Fast Filesystem
options     SOFTUPDATES     # Enable FFS soft updates support
options     UFS_ACL         # Support for access control lists
options     UFS_DIRHASH     # Improve performance on big directories
options     UFS_GJOURNAL    # Enable gjournal-based UFS journaling
options     QUOTA           # Enable disk quotas for UFS
options     MD_ROOT         # MD is a potential root device
options     NFSCL           # New Network Filesystem Client
options     NFSD            # New Network Filesystem Server
options     NFSLOCKD        # Network Lock Manager
options     NFS_ROOT        # NFS usable as /, requires NFSCL
options     MSDOSFS         # MSDOS Filesystem
options     CD9660          # ISO 9660 Filesystem
options     PROCFS          # Process filesystem (requires PSEUDOFS)
options     PSEUDOFS        # Pseudo-filesystem framework
options     GEOM_PART_GPT   # GUID Partition Tables.
options     GEOM_RAID       # Soft RAID functionality.
options     GEOM_LABEL      # Provides labelization
options     COMPAT_FREEBSD32    # Compatible with i386 binaries
options     COMPAT_FREEBSD4     # Compatible with FreeBSD4
options     COMPAT_FREEBSD5     # Compatible with FreeBSD5
options     COMPAT_FREEBSD6     # Compatible with FreeBSD6
options     COMPAT_FREEBSD7     # Compatible with FreeBSD7
options     SCSI_DELAY=5000     # Delay (in ms) before probing SCSI
options     KTRACE          # ktrace(1) support
options     STACK           # stack(9) support
options     SYSVSHM         # SYSV-style shared memory
options     SYSVMSG         # SYSV-style message queues
options     SYSVSEM         # SYSV-style semaphores
options     _KPOSIX_PRIORITY_SCHEDULING # POSIX P1003_1B real-time extensions
options     PRINTF_BUFR_SIZE=128    # Prevent printf output being interspersed.
options     KBD_INSTALL_CDEV    # install a CDEV entry in /dev
options     HWPMC_HOOKS     # Necessary kernel hooks for hwpmc(4)
options     AUDIT           # Security event auditing
options     CAPABILITY_MODE     # Capsicum capability mode
options     CAPABILITIES        # Capsicum capabilities
options     PROCDESC        # Support for process descriptors
options     MAC         # TrustedBSD MAC Framework
options     KDTRACE_FRAME       # Ensure frames are compiled in
options     KDTRACE_HOOKS       # Kernel DTrace hooks
options     DDB_CTF         # Kernel ELF linker loads CTF data
options     INCLUDE_CONFIG_FILE     # Include this file in kernel

# Debugging support.  Always need this:
options     KDB             # Enable kernel debugger support.
options     KDB_TRACE       # Print a stack trace for a panic.

# Make an SMP-capable kernel by default
options     SMP             # Symmetric MultiProcessor Kernel

# CPU frequency control
device      cpufreq

# Bus support.
device      acpi
device      pci

# ATA controllers
device      ahci        # AHCI-compatible SATA controllers
device      ata     # Legacy ATA/SATA controllers

# ATA/SCSI peripherals
device      scbus       # SCSI bus (required for ATA/SCSI)
device      da      # Direct Access (disks)
device      cd      # CD

# atkbdc0 controls both the keyboard and the PS/2 mouse
device      atkbdc      # AT keyboard controller
device      atkbd       # AT keyboard
device      psm     # PS/2 mouse

device      kbdmux      # keyboard multiplexer

device      vga     # VGA video card driver
options     VESA        # Add support for VESA BIOS Extensions (VBE)

device      splash      # Splash screen and screen saver support

# syscons is the default console driver, resembling an SCO console
device      sc
options     SC_PIXEL_MODE   # add support for the raster text mode

device      agp     # support several AGP chipsets

# PCI Ethernet NICs.
device      em      # Intel PRO/1000 Gigabit Ethernet Family
device      igb     # Intel PRO/1000 PCIE Server Gigabit Family
device      ixgbe       # Intel PRO/10GbE PCIE Ethernet Family

# Pseudo devices.
device      loop        # Network loopback
device      random      # Entropy device
#device     padlock_rng # VIA Padlock RNG
device      rdrand_rng  # Intel Bull Mountain RNG
device      ether       # Ethernet support
device      vlan        # 802.1Q VLAN support
device      tun     # Packet tunnel.
device      md      # Memory "disks"
device      gif     # IPv6 and IPv4 tunneling
device      faith       # IPv6-to-IPv4 relaying (translation)
device      firmware    # firmware assist module

# The `bpf' device enables the Berkeley Packet Filter.
# Be aware of the administrative consequences of enabling this!
# Note that 'bpf' is required for DHCP.
device      bpf     # Berkeley packet filter

# USB support
options     USB_DEBUG   # enable debug msgs
device      uhci        # UHCI PCI-USB interface
device      ohci        # OHCI PCI-USB interface
device      ehci        # EHCI PCI-USB interface (USB 2.0)
device      xhci        # XHCI PCI-USB interface (USB 3.0)
device      usb     # USB Bus (required)
device      ukbd        # Keyboard
device      umass       # Disks/Mass storage - Requires scbus and da

#
### EOF ###




Do these optimizations really make a difference for a web server ?

Lets take a look at the web server performance for our server, calomel.org before and after modifications. Keep in mind the graphs are the result of the exact same hardware, the same network, the same files and access is 100% public requests. We are only graphing successful requests (code 200) and no code 301 redirections or errors.

With Nginx you can setup the log lines to tell you how long it took to fulfill the client request and complete the data transfer to the client. We have multiple examples of how to setup this log format on our Nginx Secure Web Server page. Using the log data we can graph the nginx performance times with our Web Server Distribution Performance Perl script.

BEFORE: FreeBSD 10 defaults. The following graph displays the time in 100 millisecond increments against the number of completed object transfers for the last ten thousand (10,000) log lines. The log was collected before speed optimizations using a default FreeBSD i10 install. Keep in mind an "object" is an html, jpg, css or any other file. File sizes range from 24 kilobytes up to 350 kilobytes. On the left hand side is the tall vertical line at 0 seconds going up to 7142 objects and a few smaller lines on the bottom left. The tall line at 0 seconds tells us for the last 10,000 objects served, the web server was able to send and complete the network transfer in zero(0) seconds ( i.e. less then 100 milliseconds) 71.42% of the time. (7142/10000*100=71.42). Keep in mind calomel.org is a SSL enabled site and this time also includes the https negotiation phase.

calomel@freebsd10:  ./calomel_http_log_distribution_performance.pl

   .:.  Calomel Webserver Distribution Statistics

         Log lines: 10000, Search string(s):   
       __________________________________________________________________
  7142 |.................................................................
       |.................................................................
  6157 |_________________________________________________________________
       |.................................................................
  5172 |.................................................................
       |.................................................................
  4679 |.................................................................
       |.................................................................
  3694 |_________________________________________________________________
       |.................................................................
  2709 |.................................................................
       |.................................................................
  2216 |.................................................................
       |.................................................................
  1231 |_________________________________________________________________
       |||...............................................................
   246 ||||..............................................................
       |||||||||||||||||||||||||||||_|||_|_||_|_|_|_|_|__|_______________
 Time: |^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|
       0   0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  4.6  5.0  5.5  6.0



AFTER: FreeBSD 10 optimization. The following graph shows the results after applying the optimizations found at the beginning of this page. Compared to the previous graph, we see our FreeBSD based Nginx server is able to serve 96.6% of the objects in less then 100ms. (9669/10000*100=96.69) compared to just 71.42% before. FreeBSD and Nginx are fast and we just made them 35% faster.

calomel@freebsd10:  /storage/tools/web_server_distribution_performance.pl 

   .:.  Calomel Webserver Distribution Statistics

         Log lines: 10000, Search string(s):   
       __________________________________________________________________
  9669 |.................................................................
       |.................................................................
  8181 |.................................................................
       |.................................................................
  7685 |.................................................................
       |.................................................................
  6198 |_________________________________________________________________
       |.................................................................
  5702 |.................................................................
       |.................................................................
  4214 |.................................................................
       |.................................................................
  3718 |_________________________________________________________________
       |.................................................................
  2231 |.................................................................
       |.................................................................
  1239 |_________________________________________________________________
       |.................................................................
   247 |.................................................................
       |||_|____|________________________________________________________
 Time: |^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|
       0   0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  4.6  5.0  5.5  6.0



OpenBSD 5.x rough comparison: Here is the Web Server Distribution Performance Perl script run on our log when we were using OpenBSD. This not a completely fair comparison since we are using the ACCF network connection buffer on FreeBSD, but ACCF would only represent a variation of a second or so. OpenBSD is very secure and has the latest Pf, but it is not considered the fastest of operating systems. Notice the graph shows 1042 requests completed in 0.1 seconds (i.e. less then 200 milliseconds) which equals 10.42% out of a total of 10000. With the FreeBSD optimizations enabled the server is able to complete transactions more than 18 times faster then OpenBSD. ((9660/.1)/(1042/0.2)=18.54). We respect OpenBSD for being secure, but we have to admit it is not suited for a high speed - low latency server.

calomel@openbsd5.x:  ./calomel_http_log_distribution_performance.pl

   .:.  Calomel Webserver Distribution Performance Statistics

         Log lines: 10000, Search string(s):
       __________________________________________________________________
  1042 .|................................................................
       .|................................................................
   898 _|________________________________________________________________
       .||...............................................................
   754 .|||..............................................................
       _|||______________________________________________________________
   682 .||||.............................................................
       .||||.............................................................
   539 _||||_____________________________________________________________
       .||||..||.........................................................
   395 .||||||||.........................................................
       _|||||||||________________________________________________________
   251 .||||||||||.......................................................
       .|||||||||||......................................................
   179 _||||||||||||_____________________________________________________
       .||||||||||||.....................................................
    35 ||||||||||||||||||||....||........................................
       ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
 Time: |^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|^^^^|
       0   0.5  1.0  1.5  2.0  2.5  3.0  3.5  4.0  4.5  4.6  5.0  5.5  6.0




Do these optimizations help a firewall too ?

Indeed! Take a look at the Network Tuning and Performance Guide where we show our FreeBSD firewall passing as much as 10 gigabit of data through the machine with low latency and minimal CPU usage.





Questions, comments, or suggestions? Contact Calomel.org or