HDFS Erasure Coding for Cold Data in Hadoop 3.x

Your Hadoop cluster is running out of disk space, and the usual suspect is the default three-way replication. Every block lives on three DataNodes, so a 100 TB dataset consumes roughly 300 TB of raw capacity. That overhead is acceptable for hot tables and active ETL outputs, but it is expensive for logs, snapshots, and compliance archives that are rarely read.

Useful takeaway: In Apache Hadoop 3.x, you can store cold directories with erasure coding instead of replication. Erasure coding splits data into fragments, adds parity fragments, and distributes them across DataNodes. It can cut storage overhead from about 200 percent to roughly 50 percent, but it is not a drop-in replacement for replication everywhere.

How HDFS erasure coding differs from replication

Three-way replication stores complete copies of each block. HDFS erasure coding (EC) uses Reed-Solomon encoding to split a block into data cells and parity cells. A common policy, RS-6-3, encodes six data cells into three parity cells. Nine total cells are stored across the cluster, and any six are sufficient to reconstruct the original block.

Terms used below:

• DataNode: a node that stores HDFS blocks.
• Block: the basic unit of storage in HDFS.
• Parity cell: an encoded fragment used to rebuild missing data.
• Stripe: the group of data and parity cells created from one block.

Where erasure coding fits

EC is a good match for cold, write-once data: archive logs, compliance snapshots, and historical analytics outputs. It is a poor match for hot tables, frequently updated files, or workloads that append to existing files. EC files do not support appends or in-place modifications, so HBase-style updates and mutable log collectors are unsuitable.

A practical split is to keep active datasets on three-way replication and apply EC only to directories that are written once and read occasionally.

Worked example: enable erasure coding on an archival directory

Run these commands on a gateway node with the HDFS client installed. You need HDFS superuser privileges, typically by running as the hdfs user.

# Confirm the cluster version supports erasure coding.
hadoop version

# List available erasure coding policies.
hdfs erasurecoding -listPolicies

# Set an RS-6-3 policy on a cold directory. New files written here use EC.
sudo -u hdfs hdfs erasurecoding setPolicy -path /archival/cold-data -policy RS-6-3

# Verify the policy was applied.
hdfs erasurecoding getPolicy -path /archival/cold-data

Expected checks: hadoop version should report Hadoop 3.x or later. -listPolicies should include RS-6-3. getPolicy should return RS-6-3 for the directory.

Next, write a sample file and confirm the client can decode it:

# Placeholder: replace sample.bin with a real test file.
hdfs dfs -put sample.bin /archival/cold-data/

# Read it back and compare checksums.
hdfs dfs -get /archival/cold-data/sample.bin sample.downloaded
sha256sum sample.bin sample.downloaded

If the checksums match, the client successfully reconstructed the erasure-coded blocks. If they differ, the read path or client compatibility needs investigation before you move production data.

Trade-offs and limitations

Factor	Three-way replication	Erasure coding RS-6-3
Raw capacity for six blocks	18 block copies	9 blocks (6 data + 3 parity)
Missing cells tolerated	Depends on rack placement	Up to 3 cells
Typical read path	Read a local replica	May reconstruct missing cells over the network
File updates	Append and overwrite supported	Write-once; no appends or in-place edits
Rebuild cost after failure	Copy a replica	Higher CPU and network use to rebuild stripes

Rebuilding a failed DataNode's erasure-coded blocks consumes more CPU and network bandwidth than copying a replica. During rebuilds, cluster performance can degrade. Some older ecosystem clients may also lack full support for reading EC files, so verify client compatibility before enabling EC in production.

Verification and rollback

After enabling EC, confirm the result by reading a test file and comparing its checksum to the original. Check policy on the directory with hdfs erasurecoding getPolicy. New files written under the path inherit the policy; existing files are not converted automatically.

Because setPolicy changes behavior for new writes, rollback means removing the EC policy so new files use default replication. Existing erasure-coded files remain encoded until you copy them into a replicated directory and delete the originals.

Closing

Start with one cold directory, apply RS-6-3, and validate reads with checksums. If the read path works and rebuild overhead is acceptable, expand EC to other archival directories while keeping hot workloads on three-way replication.