Performance Tuning

Tuning ZFS for Different Workloads

ZFS is highly flexible and can be tuned to optimize performance for specific workloads. Different workloads, such as sequential write-heavy applications (e.g., backups or media streaming) versus random read-heavy applications (e.g., databases), require different performance optimizations.

For write-heavy workloads, enabling ZFS's compression feature may improve performance, as less data is written to the disks. The lz4 compression algorithm, for instance, offers a balance between compression speed and efficiency, and is often recommended for general use:

zfs set compression=lz4 mypool/mydataset

In workloads with high read demands, ensuring that the Adaptive Replacement Cache (ARC) is properly sized can significantly improve performance. For large-scale datasets, adjusting the block size with recordsize may optimize I/O performance. The default record size in ZFS is 128 KB, but this can be adjusted:

• For databases that access small blocks of data frequently, setting the recordsize to 16 KB or 8 KB might help.

zfs set recordsize=16K mypool/mydataset

• For media files or other large, sequential workloads, a larger recordsize (1 MB or more) may be beneficial.

zfs set recordsize=1M mypool/mydataset

Adjusting ARC (Adaptive Replacement Cache)

The ARC (Adaptive Replacement Cache) is a critical component of ZFS that stores frequently accessed data in memory (RAM) to reduce the need for disk access. ARC is dynamically sized but can be manually tuned depending on the available memory and the needs of the workload.

The size of the ARC can be adjusted by setting the zfs_arc_max parameter, which defines the upper limit of memory that ARC can consume. This value is typically set in /etc/modprobe.d/zfs.conf on Linux or /boot/loader.conf on FreeBSD:

echo "options zfs zfs_arc_max=8589934592" >> /etc/modprobe.d/zfs.conf   # Example for 8 GB ARC

The ARC is split into two caches:

• Most Recently Used (MRU): For data that has been accessed recently.
• Most Frequently Used (MFU): For data that has been accessed frequently over time.

ZFS will automatically adjust the balance between MRU and MFU based on usage patterns, optimizing memory utilization. For systems with limited RAM, lowering the ARC size can help free up memory for other processes. However, reducing ARC size too much may lead to performance degradation due to increased disk reads.

ZIL (ZFS Intent Log) and SLOG (Separate Log Device)

The ZFS Intent Log (ZIL) is a special part of ZFS that ensures data consistency for synchronous writes. When an application performs a synchronous write (such as a database transaction), ZFS writes the data to the ZIL before confirming to the application that the data is safely written. If a system crash occurs, ZFS uses the ZIL to recover incomplete transactions.

The Separate Log Device (SLOG) is an optional dedicated device that can be added to offload the ZIL from the primary pool. Using a fast, low-latency device like an SSD for the SLOG can significantly improve performance for workloads with high volumes of synchronous writes.

To add a SLOG device to a ZFS pool, use the following command:

zpool add mypool log /dev/sdX

This command adds a dedicated log device (/dev/sdX in this example) to the pool. When the SLOG is present, ZFS will use it to store the ZIL, reducing the latency of synchronous writes. It is important to note that the SLOG only caches data temporarily until it is committed to the main pool, so it does not increase storage capacity.

The SLOG can be particularly beneficial for applications such as databases or NFS servers, where low-latency synchronous writes are essential. However, for workloads that predominantly involve asynchronous writes (such as large file transfers), the SLOG may not provide significant performance improvements.

Special Allocation Class Vdevs

While a SLOG accelerates synchronous writes and an L2ARC extends the read cache, a special allocation class vdev accelerates the pool's metadata. When a special vdev is present, ZFS stores pool metadata — the structures consulted for directory listings, file lookups, deletions, and space accounting — on it instead of on the regular data vdevs. Placing a small mirror of SSDs or NVMe devices in front of a large hard-disk pool in this way can make the pool feel dramatically more responsive, since metadata operations no longer compete with bulk data for disk seeks.

A special vdev is added with the special keyword, either at pool creation or later. Unlike a SLOG or L2ARC, a special vdev holds the only copy of the pool's metadata, so its loss destroys the entire pool — it must always be redundant, with redundancy comparable to the data vdevs it serves:

zpool add mypool special mirror /dev/nvme0n1 /dev/nvme1n1

The special class can also store small file blocks, which hard disks handle poorly. The special_small_blocks dataset property sets a block-size threshold; any block at or below it is written to the special vdev rather than to the data vdevs:

zfs set special_small_blocks=32K mypool/mydataset

The threshold must be smaller than the dataset's recordsize, otherwise every block qualifies and all new data for the dataset lands on the special vdev. Usage of the special vdev can be monitored with zpool list -v, which reports capacity per vdev. If the special vdev fills up, nothing fails: new metadata simply spills over to the regular vdevs, and the performance benefit fades until space is freed.

Key points:

• A special vdev is pool-critical: it must be mirrored, and losing it means losing the pool. This is the opposite of L2ARC (disposable) and less forgiving than a SLOG (whose loss costs at most the last few seconds of synchronous writes).
• Sizing guidance: metadata alone typically needs well under 1% of the pool's capacity; budget considerably more when special_small_blocks is enabled, since small file blocks add up quickly.
• A special vdev can only be removed again from a pool made entirely of mirrors; in pools with RAID-Z data vdevs, adding one is a permanent decision.
• A related dedup vdev class exists to hold deduplication tables specifically, keeping DDT lookups off the data disks when deduplication is in use.

Direct I/O for NVMe Pools

All reads and writes normally pass through the ARC. On hard disks this is almost always a win, but on very fast NVMe devices the extra memory copies involved in caching can themselves become the bottleneck — and applications that maintain their own caches, such as databases, pay for caching twice. OpenZFS 2.3 adds Direct I/O support: the direct dataset property controls how ZFS treats requests that applications open with the O_DIRECT flag:

zfs set direct=standard mypool/mydataset

With standard (the default), O_DIRECT requests bypass the ARC while everything else is cached as usual. always makes all eligible I/O on the dataset bypass the cache, and disabled restores the old behavior of quietly buffering even O_DIRECT requests. To benefit, requests must be properly aligned — sized in multiples of the dataset's recordsize and memory-aligned — otherwise ZFS falls back to the buffered path.

Direct I/O is a specialist's tool: it helps on pools of fast NVMe devices serving applications that do their own caching, where it removes memory-bandwidth overhead and can raise throughput considerably. For general workloads the ARC's hit rate is worth far more than the copies it costs, so standard is the right setting for almost everyone, and always should only follow from measurement.