Lowering HDFS replication factor to one for low-traffic workloads

Answer

1. Replication factor 1 should be reserved for archival or scratch data that has an external durable copy. 2. For low‑traffic production data without such a copy, a replication factor of 2 offers a better cost‑fault‑tolerance balance. 3. Block unavailability after dropping to replication 1 is an expected operational outcome and generally not acceptable for production SLAs that require durability.

Confirmed Facts

• In Hadoop 3.x the cluster‑wide default dfs.replication is 3.
• Per‑file replication can be changed with hdfs dfs -setrep; the default applies only to new files unless overridden.
• Replication 1 keeps a single block copy; loss of the hosting DataNode, a disk failure, or corruption can render blocks permanently unavailable.
• Changing replication does not retroactively protect existing blocks; they remain at the new level.

Industry Practice & Recommendations

Many operators treat replication 1 as appropriate only for data that is either non‑critical, temporary, or has an external backup (e.g., archival to object storage). For production workloads that are accessed infrequently but still require durability, replication 2 is usually the sweet spot: it cuts storage usage by roughly 33 % compared to replication 3 while still providing a single redundant copy in case of a DataNode failure.

Practical Steps for a Low‑Traffic Production Path

1. Verify the current default and existing file replication:

   hdfs getconf -confKey dfs.replication
   hdfs fsck /path/to/data -files -blocks -locations | grep "Replication:"
   

2. If you decide to move to replication 2, run:

   hdfs dfs -setrep -R 2 /path/to/data
   

3. To switch a subset of files to replication 1 (e.g., archival sub‑folder), use:

   hdfs dfs -setrep -R 1 /path/to/data/archive
   

4. After the change, monitor block health:

   hdfs fsck /path/to/data -files -blocks -locations | grep "Missing"
   

5. Configure dfs.block.min.unavailable.percent to trigger alerts if a high percentage of blocks become unavailable. This helps detect accidental replication downgrades.

Verification & Monitoring

• Run hdfs fsck / -files -blocks -locations periodically to spot missing replicas.
• Set up a simple cron job that checks the output of dfs.block.min.unavailable.percent and alerts via email or Slack.
• Document the RPO/RTO for each dataset before altering replication to avoid accidental data loss.

Missing Diagnostic Detail

Does your cluster have an external backup strategy (e.g., off‑site object storage) that can serve as a fallback for replication 1 data? The answer will influence whether replication 1 is acceptable for your workload.