Cutting HDFS Storage Overhead in Half with Erasure Coding

Your Hadoop cluster keeps three copies of every block by default. That's fine for hot data that jobs read constantly, but for archival datasets — last year's logs, completed ETL outputs, compliance snapshots — you're paying 3× storage for data that mostly sits idle. HDFS Erasure Coding (EC), available since Hadoop 3, cuts that overhead to roughly 1.5× while still tolerating the same number of node failures. This post walks through how it works, how to enable it on a directory, and where it will hurt you if you're not careful.

Replication vs. erasure coding

Default HDFS replication stores each block three times on different DataNodes. Lose any two nodes holding a block and you still have a copy. Simple, fast to read, expensive to store.

Erasure coding takes a different approach using Reed-Solomon encoding. A file is split into stripes of data blocks, and the system computes additional parity blocks from them. With the commonly used RS-6-3 policy, every stripe has 6 data blocks and 3 parity blocks — 9 blocks total for 6 blocks' worth of data, a 1.5× overhead. Any 6 of the 9 blocks are enough to reconstruct the whole stripe, so you can still lose 3 machines without losing data. Same fault tolerance as 3× replication, half the disk.

Enabling a policy on a directory

EC is applied per directory, so you can mix replicated and erasure-coded data in one namespace. Run these commands as an HDFS administrator (typically the hdfs user) on any node with the Hadoop client configured. Assumes Hadoop 3.3 or later.

# List available policies
hdfs ec -listPolicies

# Enable the RS-6-3 policy cluster-wide (one-time, admin only)
hdfs ec -enablePolicy -policy RS-6-3-1024k

# Apply it to an archival directory
hdfs ec -setPolicy -path /archive/logs -policy RS-6-3-1024k

The 1024k in the policy name is the stripe cell size — how much data is encoded at a time. The default is a reasonable starting point. Note that setPolicy only affects files written after the policy is set; existing files stay replicated until you rewrite them (a simple hdfs dfs -cp into the directory works).

Verifying the storage savings

Don't take the 1.5× figure on faith — measure it. Before and after migrating a dataset, check physical usage:

hdfs dfs -du -s /archive/logs

-du reports logical and physical size; physical should drop from ~3× to ~1.5× the logical size once blocks are fully written and old replicas cleaned up. To confirm the encoding itself, inspect a file:

hdfs fsck /archive/logs/myfile.parquet -files -blocks -locations

You should see block groups marked as erasure coded with data and parity block locations rather than three identical replicas.

What failure recovery actually looks like

When a DataNode dies, the NameNode schedules reconstruction of the missing blocks. For EC data, a worker reads the surviving data and parity blocks of each affected stripe and decodes the missing piece. You can watch this in hdfs dfsadmin -report under pending reconstruction blocks.

The catch: reconstruction reads up to 6 blocks from 6 different machines to rebuild one block. That's substantial network traffic compared to copying a single replica. On a busy cluster, a large node failure can saturate the network. Practical mitigations: throttle concurrent reconstructions via dfs.namenode.reconstruction.max.streams in hdfs-site.xml (requires a NameNode restart or config refresh depending on version), and test a controlled DataNode shutdown on a non-production cluster first to observe the load.

Where EC is the wrong choice

EC trades CPU and network for disk. Writes are slower because parity must be computed and striped across nodes. Sequential reads are comparable to replication, but a read that hits a missing block triggers a decode on the fly. Small files are a poor fit — a file smaller than one stripe wastes parity blocks and gains little. Latency-sensitive workloads (HBase regions, Hive ACID tables) either don't support EC directories or perform badly on them; check your ecosystem tools before migrating anything they manage.

The sweet spot is large, cold, sequentially-read data: archived logs, historical partitions, backup exports. If a dataset hasn't been read in 30 days and is measured in terabytes, it's an EC candidate.

Actionable next step

Pick one archival directory this week. Enable RS-6-3, copy a representative dataset in, and compare hdfs dfs -du -s before and after. Then kill one DataNode process on your test cluster, confirm the data stays readable, and watch reconstruction traffic. If the numbers hold — and for cold data they usually do — you've found a permanent 50% discount on your archival storage.