home June 04, 2019

ZFS Raidz Performance, Capacity and Integrity


comparing speed, space and safety per raidz type

ZFS includes data integrity verification, protection against data corruption, support for high storage capacities, great performance, replication, snapshots and copy-on-write clones, and self healing which make it a natural choice for data storage.

The most asked question when talking about raids is, "Which raid is the best?" The best depends on what you are trying to achieve and what you are willing to give up. The question we should be asking ourselves is, "How important is our data ?"

Eventually you will lose a drive and depending on the raid solution will decide if data is lost. If the raid loses a single drive and all your files are gone could be a disaster. You may want to look at a safer raid configuration so the raid can lose two or more drives. The problem with higher data safety is you may have to give up speed or capacity or both.

Raids have three main benefits: Performance, Capacity and Integrity. Performance is how fast the raid will read and write data and is measured in megabytes per second as well as in milliseconds of latency. Capacity is how much does the raid hold. Integrity is how many disks can fail before all the data is lost. The problem is you may not be able to take advantage of all three benefits.

The ASCII triangle below shows all three properties. If you put your mouse cursor in the center of the triangle and move to one property, you see the other properties getting farther away. For example, raid0 is both fast and has the highest capacity, but absolutely no data integrity. Raid1 on the other hand has fantastic integrity and fast reads, but slow writes (multiple copies) and has limited capacity.

        Raids have three main advantages over using a single disk.
               Performance, capacity and data integrity.

                               capacity
                                  /\
                                 /  \
                                /    \
                               /      \ 
                  performance /________\ integrity

What are the advantages and disadvantages of each raid type ?

Be Safe, Not Sorry. When choosing a raid configuration you may look at raidz or RAID5 and see the speed and capacity benefits and decide it is a good choice. From real world experience we highly suggest NOT using raid5. It is simply not safe enough. We recommend using RAID1 mirroring or RAID6 (double parity) or even RAID7 (triple parity). The problem with raid5 is you only have one drive with parity. When one drive dies the raid5 array is degraded and you can not loose another drive before you lose the entire array; i.e. all your data is lost. What happens most of the time is one drive dies, you replace the drive and the array starts resilvering or rebuilding. This is when the other disks are being stressed and during this rebuild is when you have a very good chance (around 8%) that you you will lose another drive. For more information please take some time to read NetApp Weighs In On Disks which was written in 2007, but is still valid today since drive technology has not changed much.

Specification of the testing raid chassis and environment

All tests are run on the same day on the same machine. Our goal was to rule out variability in the hardware so that any differences in performance were the result of the ZFS raid configuration itself.

Bonnie++ benchmarks

Bonnie++ is a benchmark suite aimed at performing a number of simple tests reporting hard drive and file system performance. The benchmark test uses database type calls to a single file to simulate creation, reading, and deleting of many small files.

The ZFS pools are created with LZ4 compression disabled so the Bonnie++ test data will be written directly to disk. Bonnie++ generates a 16 gigabyte file which was chosen because it could not fit into 16 gig of RAM or the ZFS ARC. Bonnie++ is using four(4) concurrent threads to better simulate real world server loads. A scripted while loop runs bonnie three(3) times and we report the median (middle) performance metric. After each bonnie run the system sleeps for 30 seconds so the load can subside.

Bonnie++ can do asynchronous I/O, which means the local disk cache of the 4k drive is heavily utilized with a flush between ZFS commits once every 30 seconds. Since the disk cache can artificially inflate the results we choose to disable drive caches completely using Bonnie++ in synchronous test mode only. Syncing after each write will result in lower benchmark values, but the numbers will more closely resemble a server which is heavily loaded and using all of its RAM.

bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4

Quick Summery: ZFS Speeds and Capacity

The following table shows a summery of all the tests and lists the number of disks, raid types, capacity and performance metrics for easy comparison. The speeds display "w" for write, "rw" for rewrite and "r" for read and throughput is in megabytes per second.

ZFS uses variable-sized blocks of up to 1024 kilobytes. If data compression (LZJB or LZ4) is enabled, variable block sizes are used. If a block can be compressed to fit into a smaller block size, the smaller size is used on the disk to use less storage and improve IO throughput at the cost of increased CPU use for the compression and decompression operations. Take a look at our compression study further down this page.

When building a RAID it is common practice to use the "power of two plus parity" raid infrastructure to maximize parity striping, speed and well as capacity. When using ZFS the standard RAID rules may not apply, especially when LZ4 compression is active. ZFS can vary the size of the stripes on each disk and compression can make those stripes unpredictable. The current rule of thumb when making a ZFS raid is:

Spinning platter hard drive raids

The server is setup using an Avago LSI Host Bus Adapter (HBA) and not a raid card. The HBA is connected to the SAS expander using a single multilane cable to control all 24 drives. LZ4 compression is disabled and all writes are synced in real time by the testing suite, Bonnie++. We wanted to test the raid configurations against the drives themselves without LZ4 compression or extra RAM to use for ZFS ARC. You can expect your speeds to easily increase by a factor of two(2) when storing compressible data with LZ4 enabled and when adding RAM at least twice as large as the data sets being read and written.

            ZFS Raid Speed Capacity and Performance Benchmarks
                   (speeds in megabytes per second)

 1x 4TB, single drive,          3.7 TB,  w=108MB/s , rw=50MB/s  , r=204MB/s 
 2x 4TB, mirror (raid1),        3.7 TB,  w=106MB/s , rw=50MB/s  , r=488MB/s 
 2x 4TB, stripe (raid0),        7.5 TB,  w=237MB/s , rw=73MB/s  , r=434MB/s 
 3x 4TB, mirror (raid1),        3.7 TB,  w=106MB/s , rw=49MB/s  , r=589MB/s 
 3x 4TB, stripe (raid0),       11.3 TB,  w=392MB/s , rw=86MB/s  , r=474MB/s 
 3x 4TB, raidz1 (raid5),        7.5 TB,  w=225MB/s , rw=56MB/s  , r=619MB/s 
 4x 4TB, 2 striped mirrors,     7.5 TB,  w=226MB/s , rw=53MB/s  , r=644MB/s 
 4x 4TB, raidz2 (raid6),        7.5 TB,  w=204MB/s , rw=54MB/s  , r=183MB/s 
 5x 4TB, raidz1 (raid5),       15.0 TB,  w=469MB/s , rw=79MB/s  , r=598MB/s 
 5x 4TB, raidz3 (raid7),        7.5 TB,  w=116MB/s , rw=45MB/s  , r=493MB/s 
 6x 4TB, 3 striped mirrors,    11.3 TB,  w=389MB/s , rw=60MB/s  , r=655MB/s 
 6x 4TB, raidz2 (raid6),       15.0 TB,  w=429MB/s , rw=71MB/s  , r=488MB/s 
10x 4TB, 2 striped 5x raidz,   30.1 TB,  w=675MB/s , rw=109MB/s , r=1012MB/s 
11x 4TB, raidz3 (raid7),       30.2 TB,  w=552MB/s , rw=103MB/s , r=963MB/s 
12x 4TB, 6 striped mirrors,    22.6 TB,  w=643MB/s , rw=83MB/s  , r=962MB/s 
12x 4TB, 2 striped 6x raidz2,  30.1 TB,  w=638MB/s , rw=105MB/s , r=990MB/s 
12x 4TB, raidz (raid5),        41.3 TB,  w=689MB/s , rw=118MB/s , r=993MB/s 
12x 4TB, raidz2 (raid6),       37.4 TB,  w=317MB/s , rw=98MB/s  , r=1065MB/s 
12x 4TB, raidz3 (raid7),       33.6 TB,  w=452MB/s , rw=105MB/s , r=840MB/s 
22x 4TB, 2 striped 11x raidz3, 60.4 TB,  w=567MB/s , rw=162MB/s , r=1139MB/s 
23x 4TB, raidz3 (raid7),       74.9 TB,  w=440MB/s , rw=157MB/s , r=1146MB/s
24x 4TB, 12 striped mirrors,   45.2 TB,  w=696MB/s , rw=144MB/s , r=898MB/s 
24x 4TB, raidz (raid5),        86.4 TB,  w=567MB/s , rw=198MB/s , r=1304MB/s 
24x 4TB, raidz2 (raid6),       82.0 TB,  w=434MB/s , rw=189MB/s , r=1063MB/s 
24x 4TB, raidz3 (raid7),       78.1 TB,  w=405MB/s , rw=180MB/s , r=1117MB/s 
24x 4TB, striped raid0,        90.4 TB,  w=692MB/s , rw=260MB/s , r=1377MB/s 

########                                                        ###########
########  Below are the zpool commands and RAW Bonnie++ output  ###########
########                                                        ###########


###############################################################################

1x 4TB, single drive, 3.7 TB,  w=108MB/s , rw=50MB/s , r=204MB/s

root@FreeBSDzfs:  zpool create storage da0

root@FreeBSDzfs:  bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           108834  25 50598  11           204522   9 393.0   4
Latency                        1992ms    2372ms              1200ms     289ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          da0       ONLINE       0     0     0

###############################################################################

2x 4TB, mirror (raid1), 3.7 TB,  w=106MB/s , rw=50MB/s , r=488MB/s

root@FreeBSDzfs: zpool create storage mirror da0 da1

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           106132  25 50303  10           488762  23 445.7   4
Latency                        9435ms    4173ms               220ms     195ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0

###############################################################################

2x 4TB, stripe (raid0), 7.5 TB,  w=237MB/s , rw=73MB/s , r=434MB/s

root@FreeBSDzfs:  zpool create storage da0 da1

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           237933  43 73012  14           434041  20 513.1   5
Latency                         505ms    4059ms               212ms     197ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          da0       ONLINE       0     0     0
          da1       ONLINE       0     0     0

###############################################################################

3x 4TB, mirror (raid1), 3.7 TB,  w=106MB/s , rw=49MB/s , r=589MB/s

root@FreeBSDzfs: zpool create storage mirror da0 da1 da2

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           106749  24 49659  11           589971  27 457.3   5
Latency                       12593ms    4069ms               134ms     191ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0

###############################################################################

3x 4TB, stripe (raid0), 11.3 TB,  w=392MB/s , rw=86MB/s , r=474MB/s

root@FreeBSDzfs:  zpool create storage da0 da1 da2

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           392378  59 86799  16           474157  22 829.3   8
Latency                         315us    4038ms               129ms     141ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          da0       ONLINE       0     0     0
          da1       ONLINE       0     0     0
          da2       ONLINE       0     0     0

###############################################################################

3x 4TB, raidz1 (raid5), 7.5 TB,  w=225MB/s , rw=56MB/s , r=619MB/s

root@FreeBSDzfs:  zpool create storage raidz da0 da1 da2

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           225143  40 56590  11           619315  30 402.9   5
Latency                        2204ms    4052ms               896ms     177ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0

###############################################################################

4x 4TB, 2 striped mirrors, 7.5 TB,  w=226MB/s , rw=53MB/s , r=644MB/s

root@FreeBSDzfs:  zpool create storage mirror da0 da1 mirror da2 da3

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           226933  42 53878   9           644043  31 427.8   4
Latency                        1356ms    4066ms              1638ms     221ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0

###############################################################################

4x 4TB, raidz2 (raid6), 7.5 TB,  w=204MB/s , rw=54MB/s , r=183MB/s

root@FreeBSDzfs:  zpool create storage raidz2 da0 da1 da2 da3

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           204686  36 54769  13           183487   9 365.5   6
Latency                        5292ms    6356ms               449ms     233ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0

###############################################################################

5x 4TB, raidz1 (raid5), 15.0 TB,  w=469MB/s , rw=79MB/s , r=598MB/s

root@FreeBSDzfs:  zpool create storage raidz1 da0 da1 da2 da3 da4

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           469287  69 79083  15           598770  29 561.2   7
Latency                        1202us    4047ms               137ms     157ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0

###############################################################################

5x 4TB, raidz3 (raid7), 7.5 TB,  w=116MB/s , rw=45MB/s , r=493MB/s

root@FreeBSDzfs:  zpool create storage raidz3 da0 da1 da2 da3 da4

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           116395  21 45652   9           493679  25 397.7   5
Latency                       27836ms    6418ms             95509us     152ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz3-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0

###############################################################################

6x 4TB, 3 striped mirrors, 11.3 TB,  w=389MB/s , rw=60MB/s , r=655MB/s

root@FreeBSDzfs:  zpool create storage mirror da0 da1 mirror da2 da3 mirror da4 da5

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           389316  62 60493  11           655534  32 804.4   9
Latency                         549us    2473ms              1997ms     186ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0

###############################################################################

6x 4TB, raidz2 (raid6), 15.0 TB,  w=429MB/s , rw=71MB/s , r=488MB/s

root@FreeBSDzfs:  zpool create storage raidz2 da0 da1 da2 da3 da4 da5

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           429953  67 71505  14           488952  25 447.7   6
Latency                         358us    4057ms               197ms     181ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0

###############################################################################

10x 4TB, 2 striped 5x raidz, 30.1 TB,  w=675MB/s , rw=109MB/s , r=1012MB/s

root@FreeBSDzfs:  zpool create storage raidz da0 da1 da2 da3 da4 raidz da5 da6 da7 da8 da9

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           675195  93 109169  21           1012768  50 817.6   9
Latency                       11619us    4471ms             84450us     110ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
          raidz1-1  ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0

###############################################################################

11x 4TB, raidz3 (raid7), 30.2 TB,  w=552MB/s , rw=103MB/s , r=963MB/s

root@FreeBSDzfs:  zpool create storage raidz3 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           552264  79 103947  20           963511  48 545.6   8
Latency                        7373us    4045ms             84226us     144ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz3-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0

###############################################################################

12x 4TB, 6 striped mirrors, 22.6 TB,  w=643MB/s , rw=83MB/s , r=962MB/s

root@FreeBSDzfs:  zpool create storage mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror da8 da9 mirror da10 da11

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           643886  91 83717  15           962904  47  1257  13
Latency                       17335us    4040ms              1884ms     175ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
          mirror-1  ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
          mirror-2  ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
          mirror-3  ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
          mirror-4  ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
          mirror-5  ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0

###############################################################################

12x 4TB, 2 striped 6x raidz2, 30.1 TB,  w=638MB/s , rw=105MB/s , r=990MB/s

root@FreeBSDzfs:  zpool create storage raidz2 da0 da1 da2 da3 da4 da5 raidz2 da6 da7 da8 da9 da10 da11

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           638918  89 105631  21           990167  49 773.7  10
Latency                       15398us    6170ms               104ms     113ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
          raidz2-1  ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0

###############################################################################

12x 4TB, raidz (raid5), 41.3 TB,  w=689MB/s , rw=118MB/s , r=993MB/s

root@FreeBSDzfs:  zpool create storage raidz da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           689424  96 118573  23           993618  52 647.3   9
Latency                       14466us    3700ms               127ms     141ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0

###############################################################################

12x 4TB, raidz2 (raid6), 37.4 TB,  w=317MB/s , rw=98MB/s , r=1065MB/s

root@FreeBSDzfs:  zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           317741  45 98974  22           1065349  56 495.5   8
Latency                       11742ms    4062ms             90593us     150ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0

###############################################################################

12x 4TB, raidz3 (raid7), 33.6 TB,  w=452MB/s , rw=105MB/s , r=840MB/s

root@FreeBSDzfs:  zpool create storage raidz3 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           452865  66 105396  24           840136  44 476.2   7
Latency                         706us    4069ms              2050ms     165ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz3-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0

###############################################################################

22x 4TB, 2 striped 11x raidz3, 60.4 TB,  w=567MB/s , rw=162MB/s , r=1139MB/s

root@FreeBSDzfs:  zpool create storage raidz3 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 raidz3 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           567667  83 162462  32           1139088  59 770.3  13
Latency                        4581us    2700ms             78597us     116ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz3-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
          raidz3-1  ONLINE       0     0     0
            da11    ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da14    ONLINE       0     0     0
            da15    ONLINE       0     0     0
            da16    ONLINE       0     0     0
            da17    ONLINE       0     0     0
            da18    ONLINE       0     0     0
            da19    ONLINE       0     0     0
            da20    ONLINE       0     0     0
            da21    ONLINE       0     0     0

###############################################################################

23x 4TB, raidz3 (raid7), 74.9 TB,  w=440MB/s , rw=157MB/s , r=1146MB/s

root@FreeBSDzfs:  zpool create storage raidz3 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           440656  64 157066  34           1146275  76 408.8   8
Latency                       21417us    2324ms               154ms     195ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz3-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da14    ONLINE       0     0     0
            da15    ONLINE       0     0     0
            da16    ONLINE       0     0     0
            da17    ONLINE       0     0     0
            da18    ONLINE       0     0     0
            da19    ONLINE       0     0     0
            da20    ONLINE       0     0     0
            da21    ONLINE       0     0     0
            da22    ONLINE       0     0     0

###############################################################################

24x 4TB, 12 striped mirrors, 45.2 TB,  w=696MB/s , rw=144MB/s , r=898MB/s

root@FreeBSDzfs:  zpool create storage mirror da0 da1 mirror da2 da3 mirror da4 da5 mirror da6 da7 mirror da8 da9 mirror da10 da11 mirror da12 da13 mirror da14 da15 mirror da16 da17 mirror da18 da19 mirror da20 da21 mirror da22 da23

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           696733  94 144526  27           898137  51  1520  16
Latency                       17247us    2850ms              2014ms   79930us

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME         STATE     READ WRITE CKSUM
        storage      ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            da0      ONLINE       0     0     0
            da1      ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            da2      ONLINE       0     0     0
            da3      ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            da4      ONLINE       0     0     0
            da5      ONLINE       0     0     0
          mirror-3   ONLINE       0     0     0
            da6      ONLINE       0     0     0
            da7      ONLINE       0     0     0
          mirror-4   ONLINE       0     0     0
            da8      ONLINE       0     0     0
            da9      ONLINE       0     0     0
          mirror-5   ONLINE       0     0     0
            da10     ONLINE       0     0     0
            da11     ONLINE       0     0     0
          mirror-6   ONLINE       0     0     0
            da12     ONLINE       0     0     0
            da13     ONLINE       0     0     0
          mirror-7   ONLINE       0     0     0
            da14     ONLINE       0     0     0
            da15     ONLINE       0     0     0
          mirror-8   ONLINE       0     0     0
            da16     ONLINE       0     0     0
            da17     ONLINE       0     0     0
          mirror-9   ONLINE       0     0     0
            da18     ONLINE       0     0     0
            da19     ONLINE       0     0     0
          mirror-10  ONLINE       0     0     0
            da20     ONLINE       0     0     0
            da21     ONLINE       0     0     0
          mirror-11  ONLINE       0     0     0
            da22     ONLINE       0     0     0
            da23     ONLINE       0     0     0

###############################################################################

24x 4TB, raidz (raid5), 86.4 TB,  w=567MB/s , rw=198MB/s , r=1304MB/s

root@FreeBSDzfs: zpool create storage raidz da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           567164  82 198696  42           1304530  81 620.7  11
Latency                       36614us    2252ms             70871us     141ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz1-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da14    ONLINE       0     0     0
            da15    ONLINE       0     0     0
            da16    ONLINE       0     0     0
            da17    ONLINE       0     0     0
            da18    ONLINE       0     0     0
            da19    ONLINE       0     0     0
            da20    ONLINE       0     0     0
            da21    ONLINE       0     0     0
            da22    ONLINE       0     0     0
            da23    ONLINE       0     0     0

###############################################################################

24x 4TB, raidz2 (raid6), 82.0 TB,  w=434MB/s , rw=189MB/s , r=1063MB/s

root@FreeBSDzfs: zpool create storage raidz2 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           434394  63 189521  39           1063626  75 516.9  12
Latency                       11369us    2130ms             80053us     153ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz2-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da14    ONLINE       0     0     0
            da15    ONLINE       0     0     0
            da16    ONLINE       0     0     0
            da17    ONLINE       0     0     0
            da18    ONLINE       0     0     0
            da19    ONLINE       0     0     0
            da20    ONLINE       0     0     0
            da21    ONLINE       0     0     0
            da22    ONLINE       0     0     0
            da23    ONLINE       0     0     0

###############################################################################

24x 4TB, raidz3 (raid7), 78.1 TB,  w=405MB/s , rw=180MB/s , r=1117MB/s

root@FreeBSDzfs: zpool create storage raidz3 da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           405579  62 180470  39           1117221  70 592.4  13
Latency                         622ms    1959ms             94830us     187ms

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          raidz3-0  ONLINE       0     0     0
            da0     ONLINE       0     0     0
            da1     ONLINE       0     0     0
            da2     ONLINE       0     0     0
            da3     ONLINE       0     0     0
            da4     ONLINE       0     0     0
            da5     ONLINE       0     0     0
            da6     ONLINE       0     0     0
            da7     ONLINE       0     0     0
            da8     ONLINE       0     0     0
            da9     ONLINE       0     0     0
            da10    ONLINE       0     0     0
            da11    ONLINE       0     0     0
            da12    ONLINE       0     0     0
            da13    ONLINE       0     0     0
            da14    ONLINE       0     0     0
            da15    ONLINE       0     0     0
            da16    ONLINE       0     0     0
            da17    ONLINE       0     0     0
            da18    ONLINE       0     0     0
            da19    ONLINE       0     0     0
            da20    ONLINE       0     0     0
            da21    ONLINE       0     0     0
            da22    ONLINE       0     0     0
            da23    ONLINE       0     0     0

###############################################################################

24x 4TB, striped raid0, 90.4 TB,  w=692MB/s , rw=260MB/s , r=1377MB/s

root@FreeBSDzfs: zpool create storage da0 da1 da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17 da18 da19 da20 da21 da22 da23

root@FreeBSDzfs: bonnie++ -u root -r 1024 -s 16384 -d /storage -f -b -n 1 -c 4
Version  1.97       ------Sequential Output------ --Sequential Input- --Random-
Concurrency   4     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
FreeBSDzfs      16G           692351  95 260259  50           1377547  75  1921  19
Latency                       12856us    1670ms             49017us   59388us

root@FreeBSDzfs:  zpool status
  pool: storage
 state: ONLINE
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        storage     ONLINE       0     0     0
          da0       ONLINE       0     0     0
          da1       ONLINE       0     0     0
          da2       ONLINE       0     0     0
          da3       ONLINE       0     0     0
          da4       ONLINE       0     0     0
          da5       ONLINE       0     0     0
          da6       ONLINE       0     0     0
          da7       ONLINE       0     0     0
          da8       ONLINE       0     0     0
          da9       ONLINE       0     0     0
          da10      ONLINE       0     0     0
          da11      ONLINE       0     0     0
          da12      ONLINE       0     0     0
          da13      ONLINE       0     0     0
          da14      ONLINE       0     0     0
          da15      ONLINE       0     0     0
          da16      ONLINE       0     0     0
          da17      ONLINE       0     0     0
          da18      ONLINE       0     0     0
          da19      ONLINE       0     0     0
          da20      ONLINE       0     0     0
          da21      ONLINE       0     0     0
          da22      ONLINE       0     0     0
          da23      ONLINE       0     0     0

###############################################################################

Solid State (Pure SSD) raids

The 24 slot raid chassis is filled with Samsung 840 256GB SSD (MZ-7PD256BW) drives. Drives are connected through an Avago LSI 9207-8i HBA controller installed in a PCIe 16x slot. TRIM is not used and not needed due to ZFS's copy on write drive leveling.

1x 256GB  a single drive  232 gigabytes ( w= 441MB/s , rw=224MB/s , r= 506MB/s )

2x 256GB  raid0 striped   464 gigabytes ( w= 933MB/s , rw=457MB/s , r=1020MB/s )
2x 256GB  raid1 mirror    232 gigabytes ( w= 430MB/s , rw=300MB/s , r= 990MB/s )

3x 256GB  raid5, raidz1   466 gigabytes ( w= 751MB/s , rw=485MB/s , r=1427MB/s )

4x 256GB  raid6, raidz2   462 gigabytes ( w= 565MB/s , rw=442MB/s , r=1925MB/s )

5x 256GB  raid5, raidz1   931 gigabytes ( w= 817MB/s , rw=610MB/s , r=1881MB/s )
5x 256GB  raid7, raidz3   464 gigabytes ( w= 424MB/s , rw=316MB/s , r=1209MB/s )

6x 256GB  raid6, raidz2   933 gigabytes ( w= 721MB/s , rw=530MB/s , r=1754MB/s )

7x 256GB  raid7, raidz3   934 gigabytes ( w= 591MB/s , rw=436MB/s , r=1713MB/s )

9x 256GB  raid5, raidz1   1.8 terabytes ( w= 868MB/s , rw=618MB/s , r=1978MB/s )
10x 256GB raid6, raidz2   1.8 terabytes ( w= 806MB/s , rw=511MB/s , r=1730MB/s )
11x 256GB raid7, raidz3   1.8 terabytes ( w= 659MB/s , rw=448MB/s , r=1681MB/s )

17x 256GB raid5, raidz1   3.7 terabytes ( w= 874MB/s , rw=574MB/s , r=1816MB/s )
18x 256GB raid6, raidz2   3.7 terabytes ( w= 788MB/s , rw=532MB/s , r=1589MB/s )
19x 256GB raid7, raidz3   3.7 terabytes ( w= 699MB/s , rw=400MB/s , r=1183MB/s )

24x 256GB raid0 striped   5.5 terabytes ( w=1620MB/s , rw=796MB/s , r=2043MB/s )

Some thoughts on the RAID results...

Raid 6 or raidz2 is a good mix of all three properties. Good speeds, integrity and capacity. The ability to lose two drives is fine for a home or office raid where you can get to the machine in a short amount of time to put in another disk. For data center use raidz2 (raid6) is good if you have a hot spare in place. Otherwise, raidz3 (raid7) might be better for data center use as it might take an admin a week to get there to replace a disk. Of course, if you use raidz2 (raid6) with a hot spare you might as well use raid7. Raid 7 would offer the same total amount of space and allow the raid to lose ANY three drives.

We found that no matter which spinning hard drive we used, the speeds and latencies were about the same. You do not have the get the most expensive SAS drives like we used. Look for any manufacture which label their drives as RE or Raid Enabled or "For NAS systems." Some Western Digital Red 3TB (WD30EFRX) hard drives performed just as well. 7200 rpm drives are around 10 percent faster then 5400 rpm drives, but the 7200 rpm drives run hotter. We recommend staying away from power saving or ECO mode drives like the Western Digital Green (WD30EZRX) series. These green drives seem to perform poorly in a raid configuration due to their ability to spin down, go into sleep mode and long SATA timings. If you are looking for all out speed, the best performance increase you are going to get is to switch from spinning hard drives to SSDs and/or enable ZFS compression.

Reliability of the pool is directly proportional to the number of disks, quality of the hardware and re-silver time. As the number of disks increase the reliability decreases because there is a higher chance for another disk to fail. Using enterprise SAS drives which have a higher meantime between failure should increase reliability compared to using desktop based personal drives. The longer the raid takes to resilver the higher the chance is another drive will fail during the re-silver process. To deal with the chance of pool failure with more disks we should increase the amount of parity disks. Higher parity count will reduce the MTTDL, Mean Time To Data Loss.

Some Random Observations: Always use LZ4 compression and stick as much RAM as you can afford into the machine. When deciding how many drives to use, drive types do not matter, SAS or SATA are both fine. The more disks in a pool, less space lost to parity equals greater space efficiency. Hard drives die in batches; try to vary the manufacturing date of the drives or buy from different companies. In general, mirrors are better for random IOPs. IOPs do not seem to differ between raidz, raidz2 or raidz3.

To compress, or not to compress, that is the question

Yes, enable LZ4 compression. The command "zfs set compression=lz4 tank" will enable compression on the "tank" pool. The long answer is also yes, and you want to be aware of the positives and negatives to make an informed decision.

Spinning drives in general are really, really slow. The OS wastes a significant amount of time just waiting to get data to and from the hard drives. The main argument against compression on the fly is it uses CPU time when reading or writing to the array and also adds latency. If you have been using computers for a while you might remember "Stacker" from the 1990's. Ideally, we want the system to contact the drives as little as possible because they are so slow.

Compression can reduce the amount of blocks the drive needs to access and the result is faster speeds at the cost of CPU time. What you gain is less blocks read or written to for even moderately compressible files meaning you have effectively increased the areal density of the drive platters. Compression allows the drive heads to move over a smaller area to access the same amount of data.

The developers of ZFS offer lz4 compression which is disabled by default. Take note that ZFS and compression use multiple CPU cores so the more cores and the faster they are the faster compression will be. With LZ4 compression enabled, the system will use around 20% more CPU time to compress and uncompress files compared to handling raw data. On average lz4 will compress data to a maximum of 2x to 3x.

In our tests we show single drive reads go from 150MB/sec to 1,174MB/sec of compressed Bonnie++ test data with lzjb. Even compressed files like bzip2, zip, mp3, avi, mkv and webm will not waste much CPU time as lz4 is a very inexpensive compression method.

In FreeBSD 10 both lzjb and lz4 are available in ZFS. LZJB is fast, but LZ4 is faster with 500MB/sec compression and 1.5GB/sec decompression speeds. This makes LZ4 50% faster on compression and 80% on decompression compared to LZJB. In FreeBSD 9.1 you may need to use "zpool set feature@lz4_compress=enabled poolname" to enable lz4 on your pool as lz4 is not backwards compatible with non-lz4 versions of zfs.

LZ4's performance on incompressible data is equally impressive. LZ4 achieves higher performance by incorporating an "early abort" mechanism which triggers if LZ4 can not reach a minimum compression ratio of 12.5%. LZ4 is perfect for admins who would like to use compression as a default option for all pools. Enabling LZ4 on all volumes means compression will be used where appropriate, while making sure the algorithm will not waste CPU time on incompressible data and reads will not incur a latency penalty due to transparent decompression. Just incredible.

The following table shows a few raid types without compression and then with lzjb and lz4 compression enabled. Bonnie++ tests use database like files which compress fairly well. You will notice as we add more physical disks, compression makes less of an impact on performance. In fact, with lz4 compression we max out read and write speeds with only three disks in raidz (raid5). Our suggestion is to enabled compression on all arrays because the amount of throughput you gain outweighs the CPU usage and you get to compress data which can save space.

# The following Bonnie++ tests show the speeds of volumes with compression off
# and then on with compression=lzjb and compression=lz4 of well compressed data.

off   1x 2TB    a single drive  1.8 terabytes ( w=131MB/s , rw= 66MB/s , r= 150MB/s )
lzjb  1x 2TB    a single drive  1.8 terabytes ( w=445MB/s , rw=344MB/s , r=1174MB/s ) 
lz4   1x 2TB    a single drive  1.8 terabytes ( w=471MB/s , rw=351MB/s , r=1542MB/s ) 

off   1x 256GB  a single drive  232 gigabytes ( w=441MB/s , rw=224MB/s , r= 506MB/s ) SSD
lzjb  1x 256GB  a single drive  232 gigabytes ( w=510MB/s , rw=425MB/s , r=1290MB/s ) SSD

off   2x 2TB    raid1  mirror   1.8 terabytes ( w=126MB/s , rw= 79MB/s , r= 216MB/s )
lzjb  2x 2TB    raid1  mirror   1.8 terabytes ( w=461MB/s , rw=386MB/s , r=1243MB/s )
lz4   2x 2TB    raid1  mirror   1.8 terabytes ( w=398MB/s , rw=354MB/s , r=1537MB/s )

off   3x 2TB    raid5, raidz1   3.6 terabytes ( w=279MB/s , rw=131MB/s , r= 281MB/s )
lzjb  3x 2TB    raid5, raidz1   3.6 terabytes ( w=479MB/s , rw=366MB/s , r=1243MB/s )
lz4   3x 2TB    raid5, raidz1   3.6 terabytes ( w=517MB/s , rw=453MB/s , r=1587MB/s )

off   5x 2TB    raid5, raidz1   7.1 terabytes ( w=469MB/s , rw=173MB/s , r= 406MB/s )
lzjb  5x 2TB    raid5, raidz1   7.1 terabytes ( w=478MB/s , rw=392MB/s , r=1156MB/s )
lz4   5x 2TB    raid5, raidz1   7.1 terabytes ( w=516MB/s , rw=437MB/s , r=1560MB/s )

off   5x 256GB  raid5, raidz1   931 gigabytes ( w= 817MB/s , rw=610MB/s , r=1881MB/s ) SSD
lzjb  5x 256GB  raid5, raidz1   931 gigabytes ( w= 515MB/s , rw=415MB/s , r=1223MB/s ) SSD

off   7x 2TB    raid7, raidz3   7.1 terabytes ( w=393MB/s , rw=169MB/s , r= 423MB/s )
lzjb  7x 2TB    raid7, raidz3   7.1 terabytes ( w=469MB/s , rw=378MB/s , r=1127MB/s )
lz4   7x 2TB    raid7, raidz3   7.1 terabytes ( w=507MB/s , rw=436MB/s , r=1532MB/s )

off   12x 2TB   raid5, raidz1   19  terabytes ( w=521MB/s , rw=272MB/s , r= 738MB/s )
lzjb  12x 2TB   raid5, raidz1   19  terabytes ( w=487MB/s , rw=391MB/s , r=1105MB/s )
lz4   12x 2TB   raid5, raidz1   19  terabytes ( w=517MB/s , rw=441MB/s , r=1557MB/s )

off   17x 2TB   raid5, raidz1   28  terabytes ( w=468MB/s , rw=267MB/s , r= 874MB/s )
lzjb  17x 2TB   raid5, raidz1   28  terabytes ( w=478MB/s , rw=380MB/s , r=1096MB/s )
lz4   17x 2TB   raid5, raidz1   28  terabytes ( w=502MB/s , rw=430MB/s , r=1473MB/s )

off   24x 2TB   raid5, raidz1   40  terabytes ( w=528MB/s , rw=291MB/s , r= 929MB/s )
lzjb  24x 2TB   raid5, raidz1   40  terabytes ( w=478MB/s , rw=382MB/s , r=1081MB/s )
lz4   24x 2TB   raid5, raidz1   40  terabytes ( w=504MB/s , rw=431MB/s , r=1507MB/s )

off   24x 256GB raid0 stripe    5.5 terabytes ( w=1340MB/s , rw=796MB/s , r=2037MB/s ) SSD
lzjb  24x 256GB raid0 stripe    5.5 terabytes ( w=1032MB/s , rw=844MB/s , r=2597MB/s ) SSD



# The raw bonnie output  

1x 2TB   a single drive  1.8 terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           445544  78 344412  67           1174294  80 285.9   3
  Latency                        6118ms    5142ms               273ms     502ms

1x 2TB   a single drive  1.8 terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           471248  76 351900  61           1542189  88 14053 178
  Latency                        8638ms    9357ms               720ms   44143us

1x 256GB   a single drive  232 terabytes SSD
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsSSD          80G           510613  88 425359  81           1290425  86  9716 140
  Latency                         509ms     553ms               277ms   13602us

2x 2TB   raid1  mirror   1.8 terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           461530  83 386759  78           1243555  85 397.2  15
  Latency                        5031ms    4847ms               276ms     304ms

2x 2TB   raid1  mirror   1.8 terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           398572  67 354963  64           1537069  90  1157  14
  Latency                       10477ms    7119ms               795ms   76540us

3x 2TB   raid5, raidz1   3.6 terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           479200  85 366110  75           1243482  87 260.8   3
  Latency                        3580ms    3717ms               500ms     342ms

3x 2TB   raid5, raidz1   3.6 terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           517140  87 453732  82           1587063  95 899.5  10
  Latency                         471ms     627ms               325ms   70999us

5x 2TB   raid5, raidz1   7.1 terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           478755  87 392926  81           1156163  83 256.9   4
  Latency                         807ms    1269ms               338ms     417ms

5x 2TB   raid5, raidz1   7.1 terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           516294  88 437536  80           1560222  93  1190  14
  Latency                         522ms    2311ms               301ms   61362us

5x 2TB   raid5, raidz1   931 gigabytes SSD
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsSSD          80G           515770  88 415428  79           1223482  85  9288 147
  Latency                         495ms    1192ms             87411us   12334us

7x 2TB   raid7, raidz3   7.1 terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           469870  86 378324  78           1127670  81 260.9   5
  Latency                        1862ms    1055ms               210ms     371ms

7x 2TB   raid7, raidz3   7.1 terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           507960  88 436097  82           1532414  92 614.7  10
  Latency                         509ms    1576ms               187ms   61843us

12x 2TB  raid5, raidz1   19  terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           487671  87 391920  81           1105113  83 248.9   4
  Latency                         503ms    1128ms               409ms     323ms

12x 2TB  raid5, raidz1   19  terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           517168  89 441478  81           1557655  93  1335  22
  Latency                         462ms     847ms               267ms   61475us

17x 2TB  raid5, raidz1   28  terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           478964  87 380927  80           1096636  82 280.7   5
  Latency                        2051ms     802ms               179ms     363ms

17x 2TB  raid5, raidz1   28  terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           502536  88 430811  81           1473594  89  1217  14
  Latency                         489ms     726ms               261ms   58670us

24x 2TB  raid5, raidz1   40  terabytes
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           478082  87 382614  79           1081919  80 278.9   6
  Latency                         990ms    1013ms               185ms     492ms

24x 2TB  raid5, raidz1   40  terabytes
 # zfs set compression=lz4 tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsFBSD10       80G           504239  87 431002  81           1507097  90  1645  22
  Latency                         514ms     844ms             58128us   43527us

24x 2TB  raid0 stripe  5.5 terabytes SSD
 # zfs set compression=off tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           587993  97 359645  65           1040082  61  7410 108
  Latency                         109ms     827ms               577ms   13935us

24x 2TB  raid0 stripe  5.5 terabytes SSD
 # zfs set compression=lzjb tank
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G         1032588  88 844972  82           2597192  85 10210 142
  Latency                        538ms     571ms               235ms   14151us

All SATA controllers are NOT created equal

The performance of your RAID is highly dependent on your hardware and OS drivers. If you are using the motherboard SATA connectors, they are not going to perform as well as a SATA expander or a dedicated raid card. Just because the SATA port says SATA 6 and comes with fancy cables does not mean the port can move data quickly. The onboard chipsets are normally the cheapest silicon the manufacturer can get away with. This sentiment is true for most of the substandard on motherboard network ports too.

On mailing lists and forums there are posts which state ZFS is slow and unresponsive. We have shown in the previous section you can get incredible speeds out of the file system if you understand the limitations of your hardware and how to properly setup your raid. We suspect that many of the objectors of ZFS have setup their ZFS system using slow or otherwise substandard I/O subsystems.

Here we test the SATA throughput of the same physical Western Digital Black 2TB SATA6 (WD2002FAEX) spinning hard drive and Samsung 840 PRO 256GB SSD on three(3) different interfaces using FreeBSD. We look at the SATA 6 Gbit/s port of a common gaming motherboard by Asus, a Supermicro server motherboard and an LSI MegaRAID raid card. All machines have at least a six(6) core CPU and 16 gig of ram. According to Western Digital the Black series 2TB (WD2002FAEX) should be able to theoretically sustain 150MB/sec sequential read and writes. The Samsung 840 Pro is rated at 540MB/sec reads and 520MB/sec writes. Notice the throughput between motherboard based SATA ports (sata 3 versus sata 6) and a dedicated card.


1x 2TB a single drive - 1.8 terabytes - Western Digital Black 2TB (WD2002FAEX)

 Asus Sabertooth 990FX sata6 onboard ( w= 39MB/s , rw= 25MB/s , r= 91MB/s )
 SuperMicro X9SRE sata3 onboard      ( w= 31MB/s , rw= 22MB/s , r= 89MB/s )
 LSI MegaRAID 9265-8i sata6 "JBOD"   ( w=130MB/s , rw= 66MB/s , r=150MB/s )

1x 256GB a single drive - 232 gigabytes - Samsung 840 PRO 256GB (MZ-7PD256BW)

 Asus Sabertooth 990FX sata6 onboard ( w=242MB/s , rw=158MB/s , r=533MB/s )
 LSI MegaRAID 9265-8i sata6 "JBOD"   ( w=438MB/s , rw=233MB/s , r=514MB/s )


# The raw bonnie output on FreeBSD 9.1

1x 2TB   a single drive  1.8 terabytes (Asus Sabertooth 990FX on board SATA6)
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsASUS         80G           39139   11 25902  8            91764    8 122.1   2
  Latency                       4015ms     3250ms              2263ms     443ms

1x 2TB   a single drive  1.8 terabytes (Supermicro X9SRE on board SATA6)
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  zfsSuper        80G           31342   6 22420   4            89085    4 180.6   1
  Latency                       4709ms    4699ms               1263ms     568ms

1x 2TB   a single drive  1.8 terabytes (LSI MegaRAID 9265-8i JBOD SATA6)
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           130325  23 66041  11           150329   8 144.8   6
  Latency                       12251ms    5102ms              1381ms     611ms

1x 256GB a single drive - 232 gigabytes (Asus Sabertooth 990FX on board SATA6)
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           224826  42 158018  30           533094  38  3788  49
  Latency                        3736ms    1422ms               652ms   26953us

1x 256GB a single drive - 232 gigabytes (LSI MegaRAID 9265-8i JBOD SATA6)
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           438424  70 233976  41           514946  29  3807  57
  Latency                       70402us    1714ms               770ms   28608us

Want more speed out of FreeBSD ? Check out our FreeBSD Network Tuning guide where we enhance 1 gigabit and 10 gigabit network configurations.

How do I align sectors to 4K for a ZFS pool ?

By default, older hard drives have a sector size of 512 bytes. Sectors sizes changed when drives became large enough that the overhead of keeping track of sectors consumed a high percentage of storage space. Many modern drives are now specified as advanced format drives which means they have 4096 byte sector sizes (4KiB). Modern 4K drives includes all SSDs and most 2TB and above magnetic drives. You can check on the manufacture website for your drives "sector size" or "sector alignment".

ZFS will query the underlying device to understand how large the sectors are and use this information to determine the size of its dynamic width stripes. A fine idea unless the hardware lies. Today, drive hardware lies more often than not. Drives claim to have a logical sector size of 512 bytes (ashift=9 or 2^9=512) while the physical sectors are 4Kib (ashift=12 or 2^12=4096). Since the drive lied, ZFS will incorrectly make stripes aligned to 512 bytes. This means stripes will almost always be non-aligned forcing the underlying device to use internal processing time which will slightly degrade write performance. We say slightly because in practice a misaligned drive only writes less then 10% slower then a properly aligned drive. Still, every byte per second counts right?

So, we are forced to manually tell ZFS exactly how we want the sectors aligned since we can not trust the hardware to tell us the truth. The procedure is not hard, but it is a little abstract. Lets setup a single drive in the ZFS pool called "tank." Using gnop we can create .nop devices to force ZFS to align physical sectors to 4K no matter what the hard drives tell us.

# Setting up a single drive ZFS pool called "tank" which is 4K aligned

# If you already have an unaligned pool you need to
# destroy it. Backup your data if need to.
 zpool destroy tank

# A 4K gnop device needs to be created which will be chained to the ZFS pool
# This will create the "tank" pool with *.nop devices forcing ZFS to have 4K
# physical sectors.
 gnop create -S 4096 /dev/mfid0
 zpool create tank /dev/mfid0.nop

# Next, the pool needs to be exported and the temporary gnop
# devices removed
 zpool export tank
 gnop destroy /dev/mfid0.nop

# Now, the aligned pool can be imported from the "raw" devices
 zpool import tank

# All done. Query ZFS to make sure the ashift equals 12 which means this pool
# is 4K aligned. If you see "ashift: 9" then you are 512b aligned.
 zdb -C tank | grep ashift
      ashift: 12

Performance of 512b versus 4K aligned pools

So, what performance difference will I see for a properly aligned 4K sector sized ZFS pool? Truthfully, we did not see much difference with one or two spinning disks. Normally less then a 10% speed variation. With alignment you may see a slight increase in hard drive speeds and your setup will be more efficient because the hard drive itself will not have to do any sector splitting or read-modify-write operations.

The performance difference seems to also depend on the type of drives you use. Green or ECO drives perform poorly even with 4K alignment compared to RE (Raid Enabled) drives. To make things more complicated some drives are simply slower then others even from the same manufacturer or family of drives. The only way to make sure you are getting the most out of your equipment is to format the drive as 512b and 4K and do the bonnie++ test on each. If you have many drives, test each one individually and put the "fast" ones in one raid and the "slower" ones in another. Mixing fast and slow drives will make the entire setup slow. It is a pain, we know, but the individual test method works reliably.

For a a new boot drive or raid we still suggest aligning the sectors. If you have a unaligned drive and want to wipe the data and re-align, keep these numbers in mind, test your own equipment and make an informed decision.

Western Digital Black 2TB (WD2001FYYG)
 512b 1x 2TB   a single drive  1.8 terabytes ( w=130MB/s , rw= 65MB/s , r=146MB/s )
 4K   1x 2TB   a single drive  1.8 terabytes ( w=131MB/s , rw= 66MB/s , r=150MB/s )

Samsung 840 PRO 256GB (MZ-7PD256BW)
 512b 1x 256GB a single drive  232 gigabytes ( w=446MB/s , rw=232MB/s , r=520MB/s )
 4K   1x 256GB a single drive  232 gigabytes ( w=438MB/s , rw=233MB/s , r=514MB/s )

##### 512 byte UN-aligned sectors Western Digital Black 2TB
#
# zdb -C tank | grep ashift
                ashift: 9
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           130463  23 65792  11           146863   8 120.6   1
  Latency                       11645ms    4845ms              2505ms     353ms

##### 4K aligned sectors Western Digital Black 2TB
#
# zdb -C tank | grep ashift
                ashift: 12
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           131335  23 66141  11           150777   9 148.2   5
  Latency                       13651ms    5009ms              1189ms     625ms

##### 512 byte UN-aligned sectors Samsung 840 PRO 256GB
#
# zdb -C tank | grep ashift
                ashift: 9
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           446299  71 232491  40           520952  30  1977  68
  Latency                       96173us    1963ms               104ms   25309us

##### 4K aligned sectors Samsung 840 PRO 256GB
#
# zdb -C tank | grep ashift
                ashift: 12
  Version  1.96       ------Sequential Output------ --Sequential Input- --Random-
  Concurrency   1     -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
  Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
  moneyslow.com     80G           438424  70 233976  41           514946  29  3807  57
  Latency                       70402us    1714ms               770ms   28608us

Your firewall is one of the most important machines on the network. Keep the system time up to date with OpenNTPD "how to" (ntpd.conf), monitor your hardware with S.M.A.R.T. - Monitoring hard drive health and keep track of any changed files with a custom Intrusion Detection (IDS) using mtree. If you need to verify a harddrive for bad sectors check out Badblocks hard drive validation/wipe.

Questions?

Do you have a ZFS Health Script we can use ?

Take a look at our ZFS Health Check and Status script. The script checks disk and volume errors as well as the health of the pool and even when the last zpool scrub was done.

Why is ZFS import not using GPT labels ?

When importing a volume you must point ZFS to the directory with the GPT labels using the "-d" argument. For example our GPT labels start with data1-XX and under Ubuntu the device labels are in /dev/disk/by-partlabel/ .

# import pool with GPT labels on Ubuntu for example.

root@zfsRAID:~#   zpool import -d /dev/disk/by-partlabel/ data1

root@zfsRAID:~#   zpool status
  pool: data1
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Apr  8 14:36:17 2033
config:

        NAME          STATE     READ WRITE CKSUM
        data1         ONLINE       0     0     0
          raidz2-0    ONLINE       0     0     0
            data1-00  ONLINE       0     0     0
            data1-01  ONLINE       0     0     0
            data1-02  ONLINE       0     0     0
            data1-03  ONLINE       0     0     0
            data1-04  ONLINE       0     0     0
            data1-05  ONLINE       0     0     0
            data1-06  ONLINE       0     0     0
            data1-07  ONLINE       0     0     0
            data1-08  ONLINE       0     0     0
            data1-09  ONLINE       0     0     0
            data1-10  ONLINE       0     0     0
            data1-11  ONLINE       0     0     0


# If you do a straight zpool import ZFS will mount the drives with the
# raw device names which we do not want.

root@zfsRAID:~#   zpool import data1
root@zfsRAID:~#   zpool status
  pool: data1
 state: ONLINE
  scan: resilvered 0 in 0h0m with 0 errors on Mon Apr  8 14:33:58 2033
config:

        NAME                                              STATE     READ WRITE CKSUM
        data1                                             ONLINE       0     0     0
          raidz2-0                                        ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe327b8d44f-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe427c0c540-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe427c9410f-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe527d1dc47-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5c8456d9cb386-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe627e2e502-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe627eb60b9-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe727fa267a-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe82802c19f-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abe92811a6f3-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abea28205d1e-part1  ONLINE       0     0     0
            scsi-3600605b00512e8c018f5abeb282e7776-part1  ONLINE       0     0     0