HDFS Storage Overhead and Erasure Coding Trade-offs

To balance storage and CPU in Hadoop 3.x, use a 4+2 Erasure Coding policy.

The recommended balance for low-traffic workloads is to implement a 4+2 Erasure Coding (EC) policy. This reduces storage overhead from 200% (3x replication) to approximately 50%, while maintaining tolerance for two node failures. To prevent write performance degradation, you should allocate at least 1.5 to 2 virtual CPU cores per DataNode specifically for handling the parity calculations required during initial writes.

Recovery Latency: EC vs. Replication

Recovery latency in EC is significantly higher than standard replication on low-density hardware. In a replicated environment, recovery is a simple network copy from a healthy node. In EC, recovering a lost block requires reading multiple data and parity blocks, performing XOR calculations to reconstruct the missing data, and writing it to a new location. On hardware with limited compute, this CPU-intensive reconstruction process can create a bottleneck, often resulting in recovery times that are 2x to 3x slower than simple replication.

Recommended Implementation Steps

• Identify cold data: Do not apply EC to the entire cluster. Use it only for directories containing infrequently accessed data to minimize constant CPU churn.
• Set the policy: Use the HDFS command to set the policy at the directory level:

  hdfs dfs -setStoragePolicy -type ec-rs-4-2 /path/to/cold/data

• Tune Threading: Adjust dfs.datanode.ec.threads to align with your available CPU cycles to prevent the encoding tasks from starving other cluster services.
• Monitor Metrics: Use JMX to track dfs.datanode.ec.encode.time and dfs.datanode.ec.recovery.time during testing.

Assumptions and Uncertainty

This analysis assumes the cluster is running Hadoop 3.0 or higher, where EC is a native feature. The specific performance impact varies heavily based on the network bandwidth; if your network is saturated, the CPU overhead of EC may be less noticeable than the increased internal traffic required for reconstruction. We are uncertain whether your specific hardware has IOPS limits, which could further throttle the EC reconstruction speed.