Cutting HDFS Storage Overhead with Erasure Coding in Hadoop 3.x

If your HDFS cluster is filling up with data that is written once and read rarely, erasure coding (EC) is the supported way to cut storage overhead from 200% (the default 3x replication) down to roughly 50% — without reducing the number of concurrent failures you can tolerate. The trade-off: recovery and reads cost more CPU and network, and only files written after you set the policy are encoded.

How erasure coding replaces replication

With 3x replication, a 1 TB logical dataset consumes 3 TB of raw disk. Erasure coding instead splits a file into stripes, and each stripe into k data cells plus m parity cells. The default policy, RS-6-3-1024k, uses Reed-Solomon encoding with 6 data cells and 3 parity cells, each 1 MB. Any 6 of the 9 cells can reconstruct the stripe, so you tolerate up to 3 concurrent cell losses — the same fault tolerance as 3x replication — at 1.5x raw storage instead of 3x.

That 1 TB dataset now costs about 1.5 TB raw. On a large archive, this is the difference between buying more disks and not.

Worked configuration: enabling EC for a cold-data directory

EC is applied per directory, not per file, and the policy must first be enabled cluster-wide. Run these as an HDFS administrator (typically the hdfs user or a supergroup member) from any node with the Hadoop client configured:

# 1. See which policies ship with the cluster
hdfs ec -listPolicies

# 2. Enable the built-in policy (disabled by default)
hdfs ec -enablePolicy -policy RS-6-3-1024k

# 3. Create the target directory and attach the policy
hdfs dfs -mkdir -p /cold/archive
hdfs ec -setPolicy -path /cold/archive -policy RS-6-3-1024k

# 4. Confirm the policy is active
hdfs ec -getPolicy -path /cold/archive

Everything written under /cold/archive from this point on is erasure-coded. Subdirectories inherit the policy unless you override it. To revert a directory back to replication, run hdfs ec -unsetPolicy -path /cold/archive; again, this only affects new writes.

Verify the savings actually happened

Do not trust the policy listing alone — verify on disk. Write a multi-GB file to the directory, then compare logical and raw sizes:

hdfs dfs -du -s /cold/archive          # logical size
hdfs fsck /cold/archive -files -blocks # block group layout, shows EC block groups

hdfs fsck output for EC files shows block groups with 9 blocks (6 data + 3 parity) rather than replicated blocks with 3 copies each. You can also check the NameNode UI or hdfs dfsadmin -report, but remember that capacity numbers now mix replicated and EC data — see the capacity-planning note below.

Cluster sizing: the 9-DataNode floor

RS-6-3 places each of the 9 cells of a stripe on a distinct DataNode (subject to rack-aware placement). A cluster with fewer than 9 DataNodes cannot satisfy the policy; writes will be degraded or fail placement. If you run a small cluster, either add nodes or choose a narrower policy if your release ships one — but check hdfs ec -listPolicies for what is actually available rather than assuming.

Limits and common mistakes

Existing files are not converted

Setting a policy on a directory does nothing to data already there. To convert existing data you must rewrite it — typically by distcp-ing or copying it into the EC directory (or into a new EC directory and swapping). Plan for the copy time and the temporary double storage during migration.

EC is for cold, large, append-once data

HDFS files are immutable, so any rewrite re-encodes the entire file, and the EC write path (striping plus parity computation) is more expensive than replication. Applying EC to hot or frequently rewritten data is the most common mistake. Good candidates: log archives, historical partitions, backups, compliance data. Bad candidates: staging directories, intermediate job output, anything updated daily.

Small files gain little

EC operates on 1 MB cells. A file smaller than one stripe (6 MB of data cells for RS-6-3) produces mostly empty cells and minimal savings, while still paying the EC read overhead. Consolidate small files first (HAR, sequence files, or upstream compaction) if the archive is full of them.

Reads and recovery are heavier

Striped reads pull cells from multiple DataNodes, so random-read-heavy workloads see higher latency, and data locality is weaker — MapReduce or Spark jobs over EC data typically move more bytes across the network. Reconstruction after a failure requires reading surviving cells and decoding, which is far more CPU- and network-intensive than copying a replica. The failure tolerance is the same on paper; the recovery cost is not.

Capacity math changes

Once you mix EC and replicated data, a simple "used / capacity" reading from dfsadmin -report no longer maps to one overhead factor. Track EC and replicated usage separately when forecasting disk purchases.

Validate before rolling out

On a test cluster, do three things before trusting EC with real archives:

1. Confirm the policy with hdfs ec -getPolicy -path <dir> and write a multi-GB file; verify with hdfs fsck that block groups show the expected 6+3 layout.
2. Stop one DataNode and confirm the EC file stays readable and that missing cells are reconstructed. Note the recovery time compared with a replicated file of the same size.
3. Run a representative query against EC and replicated copies of the same dataset and measure the read-throughput difference. If your "cold" data is actually queried weekly, that latency number decides whether EC is worth it.

One version caveat: policy names and CLI syntax shifted between Hadoop 3.0/3.1 and later 3.x releases. Check the documentation for your exact release before scripting any of the commands above.