HDFS Erasure Coding: Trading CPU for Disk on Cold Data

If you run an HDFS cluster, you're probably paying for storage you don't need. Triple replication means every byte of cold data occupies three bytes of raw disk. For rarely accessed data, that's a 200% overhead. HDFS erasure coding cuts that to roughly 50% while keeping comparable fault tolerance — at the cost of CPU and read complexity. This post explains how it works, how to enable it, and how to verify it's actually saving you space.

Why triple replication hurts at scale

HDFS's default durability model is simple: write each block three times on three different DataNodes. For hot data, that's cheap insurance. For cold data — logs, archives, backups — it's expensive. A 10 GB file consumes 30 GB of raw storage. Multiply that by petabytes of data, and the cost becomes a budget line item.

Erasure coding basics

Erasure coding splits a file into data blocks and parity blocks, then distributes them across DataNodes. The RS-6-3 policy (Reed-Solomon, 6 data blocks, 3 parity blocks) stores 6 data blocks and 3 parity blocks. You can lose up to 3 blocks and still reconstruct the file. That's comparable to triple replication, but with far less raw storage.

A worked example

Take a 10 GB file. With triple replication, you write 30 GB. With RS-6-3-1024k, the file is split into six 1 MB data blocks and three 1 MB parity blocks. Raw storage: 9 MB instead of 30 MB. The parity blocks are distributed across different DataNodes, so you can lose up to three blocks and still recover the file.

Trade-offs and limitations

Erasure coding is not free. Reads require reading parity blocks and reconstructing missing data, which adds CPU and network load. Random access is slower. Legacy applications may not understand erasure coding and will need a fallback to replicated directories.

How to enable and verify

Enable the policy, then test with a copy and fsck:

hdfs ec -enablePolicy -policy RS-6-3-1024k
hdfs dfs -mkdir /ec-test
hdfs distcp -update hdfs://source hdfs://ec-test
hdfs fsck /ec-test -files -blocks -locations

Check the fsck output to confirm the file is stored as erasure coded blocks. Then monitor encode/decode latency via JMX metrics such as dfs.erasurecoding.encode.latency and dfs.erasurecoding.decode.latency. If latency or CPU climbs too high, your workload may not be a good fit.

Actionable closing

Start with a test directory, enable the policy, and run a small file through the pipeline. Verify with fsck and monitor the metrics. If the trade-off is acceptable, roll it out to cold data first. Keep hot data replicated.