ZFS Deduplication

Deduplication eliminates redundant copies of identical data blocks. Instead of storing the same block multiple times, the system keeps a single instance and replaces duplicates with references to it. This reduces storage consumption and can improve efficiency. The effectiveness of deduplication depends on the dataset: backups and archives often contain many duplicates and deduplicate well, while unique or already compressed data (such as media files) typically shows lower deduplication ratios.

vStor 4.13 uses ZFS 2.3.0 deduplication. Deduplication settings are managed at the pool level.

Editing Pool Deduplication Settings

1. Go to the Storage view.

2. Select the pool for which you want to change deduplication settings.

3. Scroll down to the Pool Deduplication pane.

1. Click Edit to launch the Edit Pool Deduplication dialog window.

The Edit Pool Deduplication dialog allows you to configure quota and memory handling for the deduplication table associated with a storage pool.

Deduplication Table Quota

Defines the maximum size of the deduplication table.

• No limit The deduplication table can grow without restrictions until available system resources are exhausted.

• Auto The system dynamically adjusts the table size based on available memory and workload requirements.

• Set value Allows the administrator to define a hard limit (in gigabytes) for the deduplication table. The table will not exceed the specified quota.

Tip. When selecting quota management settings:

• Use Auto when system resources are sufficient, and dynamic adjustment is preferred.

• Use Set value in environments where resource usage must be strictly controlled.

• Avoid No limit in resource-constrained systems, as this can lead to memory exhaustion.

Deduplication Table Prefetch

• Load deduplication table to memory at pool import When enabled, the deduplication table is fully preloaded into memory during pool import. This can improve performance at the cost of increased memory usage. When disabled, the table is loaded incrementally, on demand.

Tip. When selecting prefetch handling:

• Enable prefetch when faster access at pool import is required and adequate memory is available.

• Disable prefetch on systems with limited memory, or when immediate loading performance is not critical.

Pruning Deduplication Table

Pruning the deduplication table allows you to remove outdated or excess entries from the deduplication table. It helps control table size and ensures efficient use of system resources.

1. Go to the Storage view.

2. Select the pool for which you want to change deduplication settings.

3. Scroll down to the Pool Deduplication pane.

4. Click Prune deduplication table to launch the Prune Deduplication Table dialog window.

Time-based

• Delete all unique records older than the specified number of days Removes entries from the deduplication table once they exceed the defined age (in days).

Size-based

• Delete enough unique records to keep their percentage below the specified threshold Removes records until the proportion of unique (non-duplicate) entries falls under the configured threshold (%).

Tip. When managing deduplication table pruning:

• Use Time-based pruning to enforce predictable retention of deduplication records, ensuring data is kept for a minimum number of days.

• Use Size-based pruning when controlling table growth is more important than retaining records for a specific duration.

A shorter time period or stricter size threshold will reduce memory usage but may also lower deduplication efficiency. Longer retention or higher thresholds improve deduplication ratios but require more system resources.