ZFS Deep Dive | TrueNAS - Wyatt's Notes

ZFS is not a traditional filesystem. It is a combined volume manager and filesystem built from three Distinct layers:

graph TD
    A[Application / POSIX Interface] --> B[ZPL - ZFS POSIX Layer]
    B --> C[DMU - Data Management Unit]
    C --> D[SPA - Storage Pool Allocator]
    D --> E[Physical Storage - vdevs]

1. ZPL (ZFS POSIX Layer): The filesystem layer that provides POSIX-compliant semantics. Files, directories, permissions, extended attributes, and ACLs. It translates file operations into block operations.
2. DMU (Data Management Unit): The transactional layer that manages objects, blocks, and snapshots. All writes are handled as atomic transactions. The DMU also manages the ARC (Adaptive Replacement Cache).
3. SPA (Storage Pool Allocator): The lowest layer that manages physical storage. It handles vdev topology, I/O scheduling, checksumming, compression, and self-healing.

ZFS never overwrites data in place. Every write creates a new copy of the data block, and only after The new block is written and its checksum verified does ZFS update the metadata to point to the new Block. This has several consequences:

• Snapshots are instantaneous and free (initially). A snapshot is a marker in the transaction history that prevents old blocks from being freed.
• No write hole. Unlike hardware RAID, a power loss during a write cannot leave data and parity in an inconsistent state. Either the old data or the new data is referenced, never a partial update.
• Fragmentation is inevitable. Over time, as blocks are updated and freed, the pool becomes fragmented. This is the primary trade-off of copy-on-write.

Every block in ZFS is identified by its content hash (SHA-256 by default), forming a Merkle tree. The root of the Merkle tree (the block pointer) stores the checksum of its child blocks, and so on Recursively.

When ZFS reads a block, it:

1. Reads the block and its checksum from disk.
2. Recomputes the checksum of the read data.
3. Compares the computed checksum with the stored checksum.
4. If they match, the data is returned.
5. If they do not match (silent corruption), ZFS uses redundancy (mirror or parity) to reconstruct the correct data and repair the corrupted copy.

A ZFS pool (zpool) is constructed from one or more vdevs (virtual devices). The vdev is the Fundamental unit of redundancy and performance. Data is striped across vdevs for performance.

graph TD
    A[zpool] --> B[vdev 0: mirror-0]
    A --> C[vdev 1: raidz2-0]
    B --> D[disk1]
    B --> E[disk2]
    C --> F[disk3]
    C --> G[disk4]
    C --> H[disk5]
    C --> I[disk6]

RAIDZ uses dynamic stripe width. Unlike traditional RAID 5/6 where the stripe width is fixed (e.g., 4+1 for RAID 5 with 5 drives), RAIDZ varies the stripe width based on the size of the incoming Write:

• Small writes (less than one sector per data disk) are written as “full stripe” writes with variable-width padding.
• Large writes that fill the stripe exactly avoid any padding overhead.
• Medium writes may leave some sectors unused (wasted space).

This is why RAIDZ capacity is slightly less than the theoretical (N-P)/N formula, where P is the Parity count. The actual usable capacity depends on the recordsize and write patterns.

The ashift property controls the physical sector size that ZFS assumes for the drives. It must be Set at pool creation time and cannot be changed afterward.

# Mirror pool (2-way mirror)
zpool create tank mirror /dev/sda /dev/sdb

# RAIDZ2 pool with ashift=12
zpool create -o ashift=12 tank raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd

# Hybrid pool: SSD mirror for special vdev + HDD RAIDZ2 for data
zpool create -o ashift=12 tank \
  mirror /dev/nvme0n1 /dev/nvme1n1 \
  raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

A dataset (zfs filesystem) is a logical namespace within a pool. Each dataset has its own Properties, mount point, and can have its own snapshots, quotas, and compression settings.

## Create a dataset
zfs create tank/data

## Set properties
zfs set compression=lz4 tank/data
zfs set atime=off tank/data
zfs set recordsize=128K tank/data
zfs set quota=500G tank/data

# List all properties
zfs get all tank/data

The recordsize property determines the maximum block size ZFS uses for a file. ZFS uses Variable-size blocks up to this maximum. The optimal recordsize depends on the workload:

Because ZFS is copy-on-write, a snapshot is a point-in-time marker in the transaction History. Creating a snapshot is instantaneous and consumes no space initially. Space is consumed Only when blocks referenced by the snapshot are modified or deleted in the live filesystem.

The space used by a snapshot is the total size of blocks that have been modified or deleted in the Live filesystem since the snapshot was taken. This is called “written” space:

# List snapshots and their space usage
zfs list -t snapshot -o name,used,refer,written

# Check space used by a specific snapshot
zfs list -o name,used,refer tank/data@daily.2024-01-01

# Create a snapshot
zfs snapshot tank/data@daily.2024-01-01

# List snapshots
zfs list -t snapshot

# Destroy a snapshot
zfs destroy tank/data@daily.2024-01-01

# Destroy snapshots matching a pattern
zfs destroy tank/data@daily.2023-*

# Clone a snapshot (creates a writable copy)
zfs clone tank/data@daily.2024-01-01 tank/data-restore

# Promote a clone (make it independent of the snapshot)
zfs promote tank/data-restore

The ARC is ZFS”s primary read cache, stored in system RAM. It uses the Adaptive Replacement Cache Algorithm, which maintains two lists:

• MRU (Most Recently Used): Recently accessed data.
• MFU (Most Frequently Used): Frequently accessed data.

The ARC dynamically balances between these two lists, evicting from the list with lower hit rates. This performs better than a simple LRU cache for mixed workloads with both sequential and random Access patterns.

ARC sizing: The default ARC maximum is 50% of system RAM on TrueNAS. For dedicated NAS Workloads, increasing the ARC to 70–80% of RAM can significantly improve read performance for hot Datasets.

# Check ARC stats
zfs get arcstats 2>/dev/null || cat /proc/spl/kstat/zfs/arcstats

# Key metrics:
# arc_hits    — Cache hits
# arc_misses  — Cache misses
# arc_hit_ratio — Percentage of reads served from cache

The L2ARC is a secondary read cache stored on a dedicated SSD (or partition of an SSD). When the ARC Evicts data, it can write it to the L2ARC before discarding it entirely. On subsequent accesses, if The data is not in the ARC but is in the L2ARC, it can be read from the L2ARC rather than from the Slower pool disks.

L2ARC considerations:

• L2ARC is read-through, not write-through. Data is written to L2ARC only when evicted from ARC.
• The L2ARC does not speed up writes — only reads.
• L2ARC requires significant ARC space to be effective. The ARC metadata for tracking L2ARC entries consumes RAM.
• L2ARC is most effective when the working set is larger than ARC but smaller than ARC + L2ARC.

The SLOG (Separate Log) is an accelerator for synchronous writes. When a synchronous write request Arrives (from NFS, SMB sync, or a database), ZFS must ensure the data is on stable storage before Acknowledging the write. Without a SLOG, this means writing directly to the pool, which is slow for HDD-based pools.

A dedicated SLOG device ( a low-latency NVMe SSD or Intel Optane) absorbs synchronous Writes at SSD speed, then asynchronously flushes them to the pool. This dramatically improves NFS And database write performance on HDD-based pools.

A scrub reads all data in the pool and verifies checksums. If a checksum mismatch is detected (the Block is corrupted), ZFS automatically repairs it from a redundant copy (mirror or parity).

# Start a scrub
zpool scrub tank

# Check scrub status
zpool status tank

# Scrub scheduling on TrueNAS:
# Configure under Data Protection → Scrub Tasks
# Recommended: Monthly scrubs for HDD pools, Weekly for SSD pools

Scrub best practices:

• Run scrubs at off-peak hours. Scrubbing a large HDD pool can take days and significantly impacts pool performance.
• SSD pools scrub much faster (hours instead of days) due to higher throughput.
• Monitor scrub progress with zpool status. The scrub will report any errors found and repaired.
• If a scrub finds uncorrectable errors, immediately back up critical data and replace the failing drive.

A resilver rebuilds the data on a replaced drive. Unlike traditional RAID rebuilds, ZFS resilvers Only copy the actual data (not the entire disk), and they prioritize data based on its metadata Importance.

# Replace a failed drive
zpool replace tank /dev/sda /dev/sdb

# Monitor resilver progress
zpool status tank

ZFS send/receive is the native mechanism for replicating datasets between pools or systems. It works At the block level, sending only the changed blocks between two snapshots.

# Full replication (initial)
zfs send tank/data@snapshot1 | zfs recv backup/data

# Incremental replication (send only changes since snapshot1)
zfs send -i tank/data@snapshot1 tank/data@snapshot2 | zfs recv backup/data

# Replication with compression over SSH
zfs send -Rcv tank/data@snapshot1 | ssh nas2 zfs recv backup/data

# Raw send (preserves encryption and compression)
zfs send -w tank/data@snapshot1 | zfs recv backup/data

zpool status -v tank

Key fields to understand:

With modern drives (8 TB+), the probability of encountering an unrecoverable read error (URE) during A resilver approaches certainty. A RAIDZ1 pool with 8 TB drives has a resilver time of 12–24 hours. During that time, reading every block on every remaining drive means the chance of hitting a URE (and losing the pool) is non-trivial. Use RAIDZ2 (or RAIDZ3) for any pool with drives larger than 4 TB.

Deduplication maintains an in-memory hash table of every unique block. This table requires Approximately 320 bytes per unique block. A 10 TB pool with 4 TB of unique data can require 100+ GB Of RAM for the dedup table. If the system runs out of RAM and must swap, performance collapses. Only Enable dedup if your data is highly redundant (VM templates, ISO images) and you have sufficient RAM. In most cases, compression (lz4) provides better space savings with no memory cost.

While ZFS allows mixing drive sizes in a RAIDZ vdev, the pool capacity is determined by the smallest Drive in the vdev. A RAIDZ2 vdev with three 12 TB drives and one 4 TB drive will have the capacity Of four 4 TB drives. Always use identical drives within a vdev.

Once a pool is created, ashift cannot be changed. Creating a pool with ashift=9 (512 bytes) on Drives with 4 KB physical sectors causes severe read-modify-write amplification on small writes, Reducing performance by 50–80%. Always use ashift=12 or higher.

ZFS pools become fragmented over time due to the copy-on-write nature. Fragmentation above 70–80% Can significantly reduce performance, especially for random read workloads. Monitor fragmentation With zpool list -v. There is no native defragmentation tool for ZFS — the only way to defragment Is to copy the data to a new pool. Regular snapshot pruning and avoiding small random writes on HDD Pools help keep fragmentation manageable.

Mirrors are the gold standard for performance and redundancy:

# 2-way mirror (most common)
zpool create -o ashift=12 -O compression=lz4 -O atime=off tank \
  mirror /dev/sda /dev/sdb \
  mirror /dev/sdc /dev/sdd \
  mirror /dev/sde /dev/sdf

# Performance characteristics:
# Read: N × single-disk IOPS (any disk in a mirror can serve the read)
# Write: N × single-disk IOPS (writes go to all mirrors simultaneously)
# Capacity: 50% of total raw
# Fault tolerance: 1 disk per mirror vdev

Mirror pools provide the best random I/O performance because every vdev can serve reads Independently. A 6-disk mirror pool (3 mirror vdevs) can serve 3× the random IOPS of a single disk.

RAIDZ2 provides dual-parity protection at better capacity efficiency:

# RAIDZ2 with 8 drives per vdev
zpool create -o ashift=12 -O compression=lz4 -O atime=off tank \
  raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh \
  raidz2 /dev/sdi /dev/sdj /dev/sdk /dev/sdl /dev/sdm /dev/sdn /dev/sdo /dev/sdp

# Performance characteristics:
# Read: Good (reads span all data disks)
# Write: Moderate (parity calculation overhead)
# Capacity: (N-2)/N of total raw per vdev
# Fault tolerance: 2 disks per vdev

DRAID is a ZFS feature that distributes spare capacity across all drives in the pool, rather than Dedicating entire drives as hot spares:

# dRAID2 with distributed spares
zpool create -o ashift=12 tank \
  draid2:2d:8c:2s /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf /dev/sdg /dev/sdh \
  /dev/sdi /dev/sdj

# Parameters:
# 2d     = 2 data drives per stripe
# 8c     = 8 children (drives) per redundancy group
# 2s     = 2 distributed spares

DRAID provides faster resilvering than traditional RAIDZ because all drives participate in Rebuilding simultaneously.

For workloads with mixed metadata and data requirements:

# NVMe metadata vdev + HDD data vdev
zpool create -o ashift=12 tank \
  mirror /dev/nvme0n1 /dev/nvme1n1 \
  raidz2 /dev/sda /dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf

# Assign special small blocks to NVMe
zfs create -o special_small_blocks=32K tank/data

This stores metadata (directories, file attributes) on the fast NVMe vdev while data resides on the HDD vdev, dramatically improving directory listing performance.

On modern CPUs (8+ cores), lz4 compression overhead is negligible for most workloads. The I/O time Saved by writing less data to disk exceeds the CPU time spent compressing.

Disable compression only for data that is already compressed or encrypted:

# Disable compression for an existing dataset
zfs set compression=off tank/media/movies
zfs set compression=off tank/backups/encrypted
zfs set compression=off tank/software/isos

The ARC maintains five lists:

1. MRU (Most Recently Used): Ghost + active MRU lists.
2. MFU (Most Frequently Used): Ghost + active MFU lists.
3. Metadata (ARC meta): Separate cache for metadata (dnode structures, directory entries).

The replacement algorithm:

• New data enters the MRU list.
• On a cache hit, data is promoted from MRU to MFU (if accessed multiple times).
• On eviction, data moves from active to ghost list. Ghost entries remember the data’s identity but not its content.
• If a ghost entry is accessed again (cache miss → hit in ghost list), the data is fetched from disk and placed at the head of the appropriate active list.
• The ARC size is bounded by arc_max (primary cache) and arc_meta_limit (metadata cache).

Metadata (directory entries, file attributes, indirect blocks) can consume a significant portion of The ARC. The arc_meta_limit parameter controls the maximum fraction of ARC dedicated to metadata:

# Default: 1/4 of ARC
# Recommended for metadata-heavy workloads: 1/2 to 3/4

# Check current metadata usage
kstat -p zfs:0:arcstats:arc_meta_used
kstat -p zfs:0:arcstats:arc_meta_max

# The metadata-to-data ratio indicates workload characteristics:
# High metadata/data ratio → Many small files (mail server, source code, home directories)
# Low metadata/data ratio → Few large files (media, VM images, backups)

# Recommended naming convention:
tank/data@daily-2024-01-15
tank/data@weekly-2024-W03
tank/data@monthly-2024-01
tank/data@pre-upgrade-2024-01-15T10-30-00
tank/data@manual-description

# List snapshots with sorting
zfs list -t snapshot -s creation -o name,creation,used,refer

# Show differences between two snapshots
zfs diff tank/data@snap1 tank/data@snap2

# Output format:
# M       +       path      # Modified file
# M       -       path      # Deleted file
# +       +       path      # New file
# R       +       old -> new # Renamed file

# Rollback a dataset to a specific snapshot (DESTRUCTIVE: destroys all snapshots taken after)
zfs rollback tank/data@daily-2024-01-10

# Force rollback (discard changes since snapshot)
zfs rollback -rf tank/data@daily-2024-01-10

# Raw send preserves encryption without requiring the key on the receiving side
zfs send -Rwv tank/encrypted@snap1 | ssh remote zfs recv -F backup/encrypted

# The receiving system cannot read the data without the encryption key
# This is ideal for offsite backup where the remote system should not have access

# Send with resume token (saved periodically)
zfs send -Rv -t <resume-token> | ssh remote zfs recv -s backup/data

# The resume token is printed when a transfer is interrupted (Ctrl+C)
# Save it and use it to resume the transfer later

# Use pv to limit bandwidth
zfs send -Rcv tank/data@snap1 | pv --rate-limit 50m | ssh remote zfs recv -F backup/data

# 50m = 50 MB/s
# Adjust based on available bandwidth and impact on production workloads

# Replace a smaller disk with a larger one (one at a time)
zpool replace tank /dev/sda /dev/sdb-new

# After all disks in a vdev are replaced with larger disks:
# The vdev automatically expands to use the full capacity
zpool list -v

# Add a new vdev to the pool (stripes across vdevs)
zpool add tank mirror /dev/sdg /dev/sdh

# Add a cache device (L2ARC)
zpool add cache tank /dev/nvme0n1

# Add a log device (SLOG)
zpool add log tank /dev/nvme1n1p1

# Add a spare device
zpool add spare tank /dev/sdi

# Export a pool (unmount all datasets)
zpool export tank

# Import a pool
zpool import tank

# Import a pool from a specific cachefile (after disk replacement)
zpool import -c /path/to/zpool.cache tank

# Import by GUID (more reliable than by name)
zpool import <guid>

# Force import (if pool was not properly exported)
zpool import -f tank

• Sharing and Permissions — ZFS datasets are shared through protocols like SMB and NFS, connecting pool management to file access.
• Backup and Replication — ZFS snapshots are the foundation for replication-based backup strategies.
• ZFS Encryption — Encryption is configured at the dataset level, building on the ZFS pool and dataset architecture.