Cutting HDFS Storage Costs in Half with Erasure Coding

The storage overhead problem

Running a Hadoop cluster at scale means paying for raw disk capacity — often three times your logical dataset size because of the default 3x replication. A 100 TB dataset consumes 300 TB on disk. For warm or cold data that isn't accessed frequently, that 200% overhead is hard to justify.

Apache Hadoop 3.0 introduced HDFS Erasure Coding (EC) as a storage-efficiency feature. Instead of three full copies, EC splits data into blocks, computes parity blocks, and spreads them across DataNodes. The default Reed-Solomon RS-6-3 policy stores six data blocks plus three parity blocks across nine DataNodes, tolerating up to three simultaneous failures while cutting raw overhead to roughly 50%. That same 100 TB dataset now needs only ~150 TB raw.

How EC changes the math

EC applies only to files with replication factor 1 that live in a directory with an EC policy. New files written to that directory inherit the policy automatically. Existing replicated files must be converted explicitly. The write path sends six data blocks to DataNodes; the first DataNode (or a dedicated EC worker) encodes parity locally and forwards the three parity blocks. For large sequential writes the pipeline is efficient; small random writes see 10–30% higher latency than replication.

On reads, if all six data blocks are available the client reads them directly — no decode step. When blocks are missing (due to node failure or decommission), the client or a standby EC reconstructor rebuilds the missing pieces from surviving data and parity blocks. That reconstruction consumes CPU and network bandwidth, which is why EC is unsuitable for hot data or workloads with frequent random reads on degraded stripes.

Worked example: enabling RS-6-3 on a cold-data directory

Assume a cluster running Hadoop 3.1+ with at least nine healthy DataNodes. Run the following commands on a client machine with HDFS superuser permissions (or a user that has write access to the target directory).

# 1. List available EC policies to confirm RS-6-3-1024k exists
hdfs ec -listPolicies

# 2. Create a directory for EC data (if it doesn't exist)
hdfs dfs -mkdir -p /data/cold

# 3. Assign the RS-6-3 policy with 1024 KB cell size
hdfs ec -setPolicy -path /data/cold -policy RS-6-3-1024k

# 4. Verify the policy is attached
hdfs ec -getPolicy -path /data/cold

New files written to /data/cold will now use EC. To convert existing replicated files in that directory:

# Convert recursively; this is I/O-intensive — run during a maintenance window
hdfs ec -replicate -path /data/cold -policy RS-6-3-1024k

Check the EC layout of a specific file:

hdfs fsck /data/cold/large-log.parquet -files -blocks -ec

Output shows block groups, parity blocks, and the DataNodes holding each block. If you see ECPolicy: RS-6-3-1024k and nine block locations per stripe, the conversion succeeded.

Trade-offs you can't ignore

• Minimum cluster size: RS-6-3 requires at least nine DataNodes. Smaller clusters must use XOR-2-1 (three nodes) or RS-3-2 (five nodes), which offer less capacity savings.
• Small files: The default 1024 KB cell size means a 1 MB file still occupies nine blocks (six data + three parity), negating space savings. Keep small files on replicated storage.
• Reconstruction load: When a DataNode fails, EC reconstruction pulls blocks from multiple racks, increasing cross-rack traffic. Tune dfs.ec.reconstruction.threads and consider YARN node labels to isolate reconstruction workers from application workloads.
• Encryption zones: HDFS Transparent Encryption with EC is supported only from Hadoop 3.3+. On 3.0–3.2, encrypted zones fall back to replication.
• Cloud object stores: S3, ADLS, and GCS do not honor HDFS EC policies. EC only applies to on-prem HDFS or HDFS-compatible filesystems like Apache Ozone.

Operational checklist before you commit

1. Verify Hadoop version: hadoop version must show 3.0.0 or later.
2. Confirm at least nine DataNodes are healthy and rack-aware.
3. Identify cold/warm datasets (logs, historical Parquet, backup snapshots) that tolerate higher read latency on failure.
4. Schedule conversion (hdfs ec -replicate) during low-usage windows; monitor NameNode metric ECReconstructionTasks and DataNode metric ECReconstructionTimeAvg.
5. Update backup/DR tools: DistCp preserves EC policy since Hadoop 3.1, but older versions may re-replicate to 3x at the destination unless -ec flags are used.
6. Keep 3x replication for hot tables, HBase WALs, and any directory requiring low-latency random reads.

Erasure coding is a powerful lever for reducing raw storage spend, but it shifts cost from disk to CPU and network during failures. Apply it selectively, monitor reconstruction metrics, and maintain a clear directory layout that separates EC-eligible data from latency-sensitive workloads.