Question
Erasure Coding Policy Commitment in Hadoop 3.x for Low-Traffic Workloads
Neel Nea
0 reputation · 29 Mar 2026, 10:54 UTC
23.8K views1
Loading…
0 reputation · 29 Mar 2026, 10:54 UTC
The objective is to evaluate whether erasure coding delivers net storage cost reduction for a low-traffic Hadoop 3.x workload without compromising read performance or increasing reconstruction latency. HDFS erasure coding, as documented in Hadoop 3.x, reduces storage overhead from the default three-way replication to approximately 1.5x for large, infrequently accessed blocks under policies such as RS-6-3-1024k.
However, the feature's efficiency is highly dependent on file size distribution; small files receive minimal striping benefit and may be better served by HAR archival or federation consolidation. A further constraint is that erasure coding policies cannot be modified on a directory after data has been written under it without rewriting the entire dataset, making the initial policy selection a persistent commitment.
What proportion of the workload's files exceed the minimum stripe size threshold for erasure coding to achieve meaningful storage savings? How does the cluster's CPU capacity compare to its I/O capacity when erasure coding reconstruction is triggered after a node failure? Would migrating small, cold files to HAR archives or consolidating under federation provide lower operational cost than deploying erasure coding across the directory tree?
30275 reputation · 29 Mar 2026, 13:48 UTC
For a low-traffic workload, erasure coding is worth committing to only if the majority of bytes (not file count) live in files large enough to fill stripe groups — under RS-6-3-1024k that means files comfortably above ~6 MB. If the workload is dominated by small files, EC delivers little storage saving, and HAR archival or directory consolidation is usually the cheaper operational path. Because an EC policy applies only to files written after it is set, treat the initial policy choice and directory layout as a one-way commitment: changing your mind means rewriting the dataset.
This is the gating metric, and only you can measure it. A logical stripe group under RS-6-3-1024k spans 6 data cells × 1024 KB ≈ 6 MB. Files well below that still allocate parity for partially filled groups, so the theoretical ~1.5x overhead (versus 3x replication) shrinks rapidly. Compute a file-size histogram weighted by bytes — via the Offline Image Viewer on a fsimage, or a recursive hdfs dfs -du pass — and check what share of total bytes sits above a few megabytes. If most bytes are in small files, stop here: EC is the wrong tool.
Reconstruction after a node or disk loss reads surviving cells from multiple nodes and recomputes the missing ones. With the native Intel ISA-L codec this is fast; with the pure-Java fallback it is CPU-hungry. For a low-traffic workload the steady-state read penalty is usually acceptable, but a reconstruction storm on a cluster with weak CPUs or already-saturated I/O can degrade everything else. Verify which codec your deployment actually loads, and benchmark by stopping one DataNode against a test EC directory while watching recovery time and load.
These solve different problems. EC reduces disk overhead for large, cold files. HAR archives reduce NameNode object overhead for many small files but do not reduce replication overhead. If your cost driver is small-file count, HAR or consolidation wins operationally; if it is raw disk on large cold files, EC wins. Mixed workloads argue for splitting directories by file-size profile and applying EC only where it pays.
hadoophdfs ec -listPolicies
hdfs ec -getPolicy -path /data/cold
hdfs dfs -du -h /data/coldThe 1.5x figure assumes fully filled stripes. Codec behavior and available policies vary across Hadoop 3.x maintenance releases — confirm against your deployed version before committing a directory tree.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.