HDFS Erasure Coding for Cold Data: An Architecture Decision Guide

The Problem: Paying 3x for Data You Rarely Read

Cold datasets that are immutable and accessed only during audits or batch backfills still consume full replication overhead in HDFS. A directory with the default replication factor of three stores three copies of every block, even when the data is frozen. HDFS erasure coding offers a controlled alternative: split each block into data and parity shards so the cluster can tolerate failures with less than 3x storage overhead. The trade-off is higher read latency and reconstruction cost. This note covers the requirements, a minimal design, trust boundaries, operational checks, and the conditions that should trigger a redesign.

Requirements and Exclusions

Erasure coding is not a general replacement for replication. It fits immutable cold datasets where you can accept higher read latency and the CPU/network cost of reconstruction. Exclude low-latency random access, active databases, append-heavy logs, MapReduce shuffle, and small files under one block. Files smaller than a single block do not benefit from sharding and may follow inefficient paths depending on the Hadoop version.

Smallest Suitable Design: One EC Policy for Cold Directories

Start with a single erasure-coding policy applied to cold directories only. A common choice is RS-6-3 (Reed-Solomon, 6 data shards, 3 parity shards), which tolerates the loss of any three shards. Keep the default replication factor of 3 for /tmp, active /user directories, HBase write-ahead logs, MapReduce shuffle, and hot tables. Apply the policy when you create the directory; do not retroactively rewrite production data without a staging migration.

Configuration Example

On an HDFS client host with HDFS superuser or directory-owner privileges, enable the policy for a new cold directory:

hdfs ec -enablePolicy -policy RS-6-3 -path /cold/archive

Verify the policy is attached:

hdfs storagepolicies -get -path /cold/archive

Expected check: the output lists the EC policy name and storage policy. Files written to this directory after the command will be stored as erasure-coded block groups. Risk: enabling a policy on a directory does not convert existing files, and append behavior for EC files varies by Hadoop release—verify against your installed distribution before writing append-heavy workloads.

Dimension	3x Replication	RS-6-3 Erasure Coding
Storage overhead	3x	~1.5x (9 total shards)
Failure tolerance	2 DataNodes	3 shards
Read latency	Low	Higher (reconstruction)
Append / random write	Supported	Version-dependent, inefficient
Best fit	Hot data, small files, active DBs	Immutable cold scans

Trust and Data Boundaries

The NameNode remains the authority for permissions. It enforces ACLs and POSIX permissions when a client opens a file. DataNodes store shards but cannot reconstruct a block without enough shards from the parity group. However, a compromised DataNode can still supply shard data to an authorized reader, so shard-level isolation is not a security boundary. Use Kerberos for RPC authentication, TLS for data-transfer encryption, and disk encryption on DataNodes as separate controls.

Operational Checks and Diagnostics

Monitor the EC policy list, block health, under-replicated blocks, the reconstruction queue, DataNode disk and network utilization, CPU, rack placement, and NameNode metadata latency. Alert when reconstruction lags or when parity groups span insufficient racks.

Run fsck on the HDFS client to compare replicated versus EC block counts and identify under-replicated blocks:

hdfs fsck /cold/archive -files -blocks -locations

Expected check: blocks under the cold directory show EC block groups with shard locations across racks. Compare under-replicated counts before and after enabling the policy using the NameNode web UI. Risk: fsck output can be large on big directories; limit the path or use the UI for aggregate counts.

Failure Modes

• Losing up to the parity count of DataNodes is tolerated. Losing more causes block unavailability until enough shards are recovered.
• Rack or network failures can trigger reconstruction storms that saturate DataNode CPU and network links.
• Small or append-heavy files create inefficient one-block or non-EC paths.
• The NameNode remains a metadata bottleneck unless HA or federation is used; erasure coding reduces storage overhead but does not solve namespace scaling.

Conditions That Change the Design

Move away from EC when reads become frequent or latency-sensitive, when the workload adds appends or random writes, when the cluster lacks enough racks or DataNodes to place parity groups, when compliance demands full replicas per site, or when reconstruction CPU and network cost exceed the storage savings. In those cases, revert to replication or move the data to an archive/object tier.

Verification and Limitations

Test on an isolated cluster before production. Create an EC directory, write a multi-block file, stop one DataNode, confirm reads succeed and blocks return to healthy after restart. Then stop parity-count-plus-one DataNodes to confirm expected unavailability. Benchmark representative cold scans on EC and replicated directories with identical hardware and network load; record latency, throughput, CPU, and reconstruction impact. Validate Kerberos/SSL, ACLs, DataNode alerts, and separate backup procedures for namespace metadata versus block data.

Limitations: EC policies are not retroactive. Append support and behavior are version-dependent. Default policies, CLI names, and reconstruction defaults vary across Hadoop 3.x releases, so verify against your installed distribution. Erasure coding increases read amplification and reconstruction CPU; do not enable it for HBase, active tables, shuffle, logs, or any low-latency workload.