Lowering HDFS replication factor to one for low-traffic workloads
0 reputation · 28 Mar 2024, 21:48 UTC
The goal is to reduce HDFS storage cost for a low-traffic workload by lowering the replication factor from the default of three to one.
In Apache Hadoop 3.x, the default replication factor is typically three. Setting it to one retains only a single block copy, so a DataNode failure or disk error can make affected blocks unavailable. The NameNode may log block health warnings when it cannot locate a readable replica during a read, but the exact message format varies by version and component and should not be treated as a fixed documented string without version-specific confirmation.
It is unresolved whether the storage savings justify the durability risk for infrequently accessed production data, or whether replication factor one should be reserved for archival workloads with external copies.
Should replication factor one be limited to archival data only? What replication factor balances cost and fault tolerance for low-traffic production workloads? Is block unavailability after reducing replication an acceptable operational outcome?