HDFS Erasure Coding Cuts Storage for Cold Data, Not for Hot Workloads

The storage tax on cold Hadoop data

HDFS defaults to three replicas of every block. A 100 TB archive therefore reserves roughly 300 TB of cluster disk. For actively queried tables, that redundancy protects availability. For cold data—compliance logs, raw event dumps, frozen ML datasets—the same 3x tax is expensive and rarely justified by access patterns.

HDFS Erasure Coding (EC) offers an alternative. Instead of storing full copies, EC splits an object into data fragments and computes parity fragments using Reed-Solomon coding. With an RS-6-3 policy, six data fragments become nine total fragments; any six can reconstruct the original data. The nominal storage overhead falls from 200 percent under 3x replication to 50 percent under RS-6-3.

EC is not magic. Reconstruction requires reading multiple fragments across the network, so rebuilds are slower and more CPU-intensive than copying a replica. It is best suited to write-once, read-infrequently workloads.

How an EC policy applies in HDFS

An EC policy is tied to a directory. Files written into that directory after the policy is set are stored as erasure-coded stripes. Each stripe spreads k data fragments and m parity fragments across distinct DataNodes. For RS-6-3, that means at least nine DataNodes are needed so fragments can sit on separate hosts; otherwise the policy cannot place blocks safely.

Existing files do not change format automatically. If you set a policy on a directory that already contains 3x replicated files, those files keep their original replication until they are rewritten—typically by copying them to a new EC-enabled directory with distcp or a Spark job.

Confirm EC is available

On a node with HDFS client access and read permission for the configuration directory, inspect the NameNode configuration:

grep -n "ec.enabled" /etc/hadoop/conf/hdfs-site.xml

Property names and defaults differ across Hadoop distributions. If EC is not enabled, changing the NameNode configuration requires a restart; coordinate with your cluster operator before editing production files.

Apply a policy to a directory

Run as the HDFS superuser or a user with write permission on the target directory:

hdfs ec -setPolicy -policy RS-6-3-1024k -path /data/archive

RS-6-3-1024k denotes six data fragments, three parity fragments, and a 1 MiB cell size. New files under /data/archive use this layout. To remove the policy for future files:

hdfs ec -unsetPolicy -path /data/archive

Unsetting the policy does not convert already-erasure-coded files back to replication. Those files must be rewritten to a 3x replicated directory.

Worked example: converting an archive directory

Suppose your cluster runs Hadoop 3.x and has at least nine DataNodes. You want to move 50 TB of immutable Parquet files from /data/raw to an EC-managed directory.

1. Create the target directory and set the EC policy:

hdfs dfs -mkdir -p /data/archive-ec
hdfs ec -setPolicy -policy RS-6-3-1024k -path /data/archive-ec

1. Rewrite the data. Existing files under /data/raw keep 3x replication:

hadoop distcp /data/raw /data/archive-ec

Distcp creates new files under the destination directory, so they inherit the EC policy.

1. Inspect the block layout rather than relying on logical size alone:

hdfs fsck /data/archive-ec -files -blocks -locations

Look for blocks spread across different DataNodes, consistent with a stripe layout. hdfs dfs -du -h reports the logical size as seen by the client and is not, by itself, a measurement of physical disk consumption across DataNodes. Compare DataNode disk usage or NameNode capacity reports before and after the rewrite to measure actual savings.

Trade-offs and hard limits

• Slower recovery. Rebuilding one missing fragment requires reading the remaining fragments across the network. Expect longer rebuild times than replica-based recovery, especially on congested links.
• Write and CPU cost. Parity calculation adds CPU load on writes. Avoid EC for frequently appended or overwritten files.
• Write-once semantics. EC files generally do not support random writes, appends, or truncates. Active Hive tables, HBase write-ahead logs, and mutable databases are poor candidates.
• Small-file penalty. Each fragment consumes a block. A small file still occupies nine blocks under RS-6-3 plus metadata, so EC can waste space on directories full of tiny files.
• Minimum cluster size. RS-6-3 needs at least nine DataNodes to place fragments on distinct hosts. Smaller clusters cannot safely use the policy.

Decide before you roll out

Apply EC to directories that are write-once, read infrequently, and contain files larger than a few hundred megabytes. Good candidates include logs older than 90 days, regulatory archives, and frozen training datasets. Start with one non-critical directory, observe rebuild behavior during a planned node failure or decommission, and compare physical disk usage against your 3x baseline before expanding.

EC reduces storage cost, not operational complexity. Keep at least one three-times-replicated copy of data that is truly irreplaceable, and plan for the extra network and CPU load that recovery demands.