Using HDFS Erasure Coding to Cut Storage Costs in Hadoop 3.x
Learn how to enable HDFS erasure coding in Hadoop 3.x to cut storage costs for cold data, with a step‑by‑step example and trade‑offs.
Loading…
ReadMeFeed / Community knowledge
#hdfs
Real questions. Useful conversations. Find the people who know your stack.
hadoopCutting HDFS Storage Costs: When to Swap 3x Replication for Erasure Coding
Stop paying the 300% storage tax. Learn how to use HDFS Erasure Coding to reduce storage overhead while maintaining data durability for cold datasets.
Minimal Rack‑Aware HDFS Design – Architecture & Ops Guide
Deploy a minimal HDFS cluster that uses rack‑aware replication to guarantee data durability and fault tolerance. The article covers required configuration, trust boundaries, operational checks, failure modes, and when to scale or add security features.
HDFS directory structure and Hive Metastore synchronization for repeatable development environments
Establishing a repeatable data development environment requires strict synchronization between the HDFS directory hierarchy and the Hive Metastore metadata. While Hive-managed tables automate the lifecycle of underlying files, maintaining consistency across different environment-lifecycles presents challenges when DDL scripts are versioned independently of t
Lowering HDFS replication factor to one for low-traffic workloads
The goal is to reduce HDFS storage cost for a low-traffic workload by lowering the replication factor from the default of three to one. In Apache Hadoop 3.x, the default replication factor is typically three. Setting it to one retains only a single block copy, so a DataNode failure or disk error can make affected blocks unavailable. The NameNode may log bloc
Does HDFS enforce authorization checks when dfs.permissions.enabled is false on a Kerberized cluster?
We are evaluating a multi-tenant HDFS deployment where Kerberos authentication is required for all client connections, but the cluster administrator is considering setting dfs.permissions.enabled to false to simplify namespace management. The concern is whether disabling the POSIX-style permission model removes all authorization barriers, effectively allowin
HDFS snapshot restore across Hadoop 3.x version boundaries: verifying recovered block integrity
HDFS snapshots in Hadoop 3.x are read-only, point-in-time namespace copies managed by the NameNode, and they are a common restore source following version transitions or directory-level data loss. The goal is to restore a snapshotted directory and then confirm that the recovered blocks match the state the snapshot recorded, without relying solely on NameNode
Federation Router Continuity vs DistCp Maintenance Window for Small-Cluster HDFS Migration
A small Hadoop application needs to move to a new cluster without a service outage, and the team is weighing two documented migration paths. HDFS Router-based Federation preserves a single filesystem view while underlying NameNodes are replaced, avoiding a hard client cutover. DistCp can replicate the dataset to a target cluster while the source stays live,
HDFS snapshot diffing and NameNode RPC pressure during small file migration
Migrating data between Hadoop clusters without downtime often relies on HDFS snapshots and incremental copies via distcopy. While this approach aims to maintain point-in-time consistency without locking source directories, the performance of the metadata comparison remains a concern for specific workloads. When the source directory contains millions of small