Replication

ZFS supports replication, allowing datasets to be copied to remote systems for backup, disaster recovery, or load balancing. Replication can be performed either as a full copy or incrementally, transferring only the changes made since the last replication. ZFS replication is efficient and can work with snapshots to provide point-in-time copies of data across systems.

Setting Up and Managing Replication

To replicate a ZFS dataset, snapshots are used to capture the current state of the dataset. These snapshots are then sent to another system or disk using the zfs send and zfs receive commands. Replication can be performed over a network or to a local disk.

First, create a snapshot of the dataset:

$ sudo zfs snapshot mypool/mydataset@snapshotname

Once the snapshot is created, it can be replicated to another ZFS system. If replicating over the network, SSH is typically used to securely transfer the data. The following command sends the snapshot to the remote system remotesystem and receives it into the remotepool pool on that system:

$ sudo zfs send mypool/mydataset@snapshotname | ssh remotesystem sudo zfs receive remotepool/mydataset

This command transfers the entire snapshot of mypool/mydataset to remotepool/mydataset on the remote system. This is a full replication, meaning the entire dataset is copied.

Managing replication involves regularly creating snapshots and sending incremental changes to the remote system. For recurring replication, a cron job or external scheduling system can be used to automate snapshot creation and sending.

To view a list of snapshots for managing replication, use:

$ sudo zfs list -t snapshot

This command will display all snapshots available for replication.

Incremental vs. Full Replication

Full Replication involves transferring the entire dataset to the target system. This is done when there is no prior data on the remote system or when starting a new replication relationship. Full replication can be useful for the initial transfer of large datasets but may require significant time and resources.

$ sudo zfs send mypool/mydataset@snapshotname | ssh remotesystem sudo zfs receive remotepool/mydataset

This command transfers the full dataset to the target system.

Incremental Replication only sends the changes (or "diffs") between two snapshots, reducing the amount of data that needs to be transferred. This is more efficient when keeping the remote system synchronized with the local dataset after the initial full replication has been completed. Incremental replication requires two snapshots: one to mark the starting point (the previous replication) and one for the current state.

For example, if snapshot1 is the previous snapshot and snapshot2 is the most recent snapshot, the following command sends only the changes between these two snapshots:

$ sudo zfs send -i mypool/mydataset@snapshot1 mypool/mydataset@snapshot2 | ssh remotesystem sudo zfs receive remotepool/mydataset

This transfers only the differences between snapshot1 and snapshot2, making incremental replication much faster and more bandwidth-efficient compared to full replication.

Key Differences:

• Full Replication: Transfers the entire dataset. Useful for initial replication or when a complete copy is needed.

• Incremental Replication: Transfers only the changes between two snapshots. More efficient for regular backups or ongoing synchronization.

Both full and incremental replication can be automated using scripts or scheduling tools, allowing ZFS replication to serve as an efficient backup solution or disaster recovery strategy.

Bookmarks and Raw Sends

Two send options solve problems that come up as replication setups mature: bookmarks remove the need to keep old snapshots on the source just to serve as incremental starting points, and raw sends allow encrypted datasets to be replicated to systems that are never given the encryption key.

Incremental Sends with Bookmarks

An incremental send normally requires the source system to retain the previous snapshot: zfs send -i @monday only works while @monday still exists on the source, and a retained snapshot holds onto every block it references, which can consume significant space on a busy dataset. A bookmark solves this. Created from an existing snapshot, it records only the point in time the snapshot represents — not the data — so it occupies almost no space:

$ sudo zfs bookmark mypool/mydataset@monday mypool/mydataset#monday

Once the snapshot has been replicated and bookmarked, it can be destroyed on the source. The bookmark (written with # instead of @) then serves as the starting point for the next incremental send:

$ sudo zfs send -i mypool/mydataset#monday mypool/mydataset@tuesday | ssh remotesystem sudo zfs receive remotepool/mydataset

This works because the receiving system still holds the real @monday snapshot from the previous replication; the bookmark only tells zfs send which changes to include. Existing bookmarks are listed with:

$ sudo zfs list -t bookmark

Raw Sends for Encrypted Datasets

When an encrypted dataset is sent normally, ZFS decrypts each block as it reads it, so the stream — and the receiving system — contains plaintext. The receiving system must be trusted, and the dataset's key must be loaded on the source for the whole transfer. A raw send, requested with -w, instead transmits the blocks exactly as they are stored on disk: encrypted data stays encrypted (and compressed data stays compressed) from end to end:

$ sudo zfs send -w mypool/securedata@monday | ssh backupserver sudo zfs receive backuppool/securedata

The backup server stores only ciphertext and never needs the encryption key. It cannot mount or read the received dataset; it simply holds it, and incremental raw sends keep it up to date. If the source system is lost, the dataset can be sent back, and the key loaded with zfs load-key on the rebuilt system restores access — the key never has to leave the systems that are trusted with it.

Key points:

• A bookmark can only serve as the source of an incremental send. It cannot be rolled back to, cloned, or sent on its own, and the receiving side must still hold the matching snapshot.
• Once a dataset has been received from a raw send, all subsequent incremental sends to that copy must also be raw; raw and non-raw streams cannot be mixed on the same target.
• For unencrypted datasets, the -c option provides the related benefit of sending compressed blocks as-is, saving bandwidth without involving encryption.