Scaling HDFS Namespace Capacity with Hadoop Federation and ViewFS

The primary bottleneck in a growing Hadoop cluster is often the memory limit of the NameNode. Because the NameNode stores the entire directory structure (file metadata and block locations) in RAM, a cluster with billions of small files will eventually exhaust available memory, regardless of how much disk space is available on DataNodes.

HDFS Federation solves this by allowing multiple NameNodes to operate in tandem, each managing its own independent namespace (called a nameservice) while sharing a common pool of DataNodes. To make this complexity transparent to users, ViewFS provides a unified file system view that routes client requests to the correct nameservice based on directory prefixes.

Desired Outcome

After completing this guide, you will have a federated HDFS cluster where:

• Two or more nameservices (e.g., ns1 and ns2) each manage a distinct namespace with their own HA NameNode pairs.
• Clients access a single logical mount point (e.g., viewfs://cluster/) and are transparently routed to the appropriate nameservice.
• DataNodes serve blocks for all nameservices without duplication.

Prerequisites

• An existing functional HDFS cluster with High Availability (HA) already configured for each nameservice (required for production).
• A Quorum Journal Manager (QJM) setup per nameservice — each nameservice needs its own JournalNode quorum (minimum 3 JournalNodes).
• Administrative permissions to modify HADOOP_HOME/etc/hadoop/ configuration files across all nodes.
• Network connectivity between all NameNodes, JournalNodes, and shared DataNodes.
• Hadoop 3.x (this guide reflects 3.3.x behavior; 2.x federation differs significantly).

Step 1: Define Nameservices in hdfs-site.xml

On all cluster nodes (NameNodes, DataNodes, JournalNodes, and client gateways), edit hdfs-site.xml to declare the federation. Federation is enabled implicitly by listing multiple nameservices in dfs.nameservices.

<property>
  <name>dfs.nameservices</name>
  <value>ns1,ns2</value>
  <description>Comma-separated list of nameservices in this federation</description>
</property>

<!-- HA configuration for ns1 -->
<property>
  <name>dfs.ha.namenodes.ns1</name>
  <value>nn1,nn2</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.ns1.nn1</name>
  <value>nn1-host:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.ns1.nn2</name>
  <value>nn2-host:8020</value>
</property>
<property>
  <name>dfs.namenode.http-address.ns1.nn1</name>
  <value>nn1-host:9870</value>
</property>
<property>
  <name>dfs.namenode.http-address.ns1.nn2</name>
  <value>nn2-host:9870</value>
</property>

<!-- HA configuration for ns2 -->
<property>
  <name>dfs.ha.namenodes.ns2</name>
  <value>nn3,nn4</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.ns2.nn3</name>
  <value>nn3-host:8020</value>
</property>
<property>
  <name>dfs.namenode.rpc-address.ns2.nn4</name>
  <value>nn4-host:8020</value>
</property>
<property>
  <name>dfs.namenode.http-address.ns2.nn3</name>
  <value>nn3-host:9870</value>
</property>
<property>
  <name>dfs.namenode.http-address.ns2.nn4</name>
  <value>nn4-host:9870</value>
</property>

<!-- QJM configuration per nameservice (each needs its own quorum) -->
<property>
  <name>dfs.namenode.shared.edits.dir.ns1</name>
  <value>qjournal://jn1:8485;jn2:8485;jn3:8485/ns1</value>
</property>
<property>
  <name>dfs.namenode.shared.edits.dir.ns2</name>
  <value>qjournal://jn1:8485;jn2:8485;jn3:8485/ns2</value>
</property>

<!-- Client failover proxy -->
<property>
  <name>dfs.client.failover.proxy.provider.ns1</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>
<property>
  <name>dfs.client.failover.proxy.provider.ns2</name>
  <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
</property>

Each nameservice (ns1, ns2) has its own HA pair and its own QJM quorum. The JournalNodes can be shared physically but must maintain separate edit log directories (the /ns1 and /ns2 suffixes above).

Step 2: Configure DataNodes for All Nameservices

On each DataNode, the same hdfs-site.xml with the full nameservice/HA definitions above is sufficient. DataNodes discover all NameNodes via the dfs.namenode.rpc-address.. properties and will register with each nameservice, creating a separate block pool (e.g., BP--, BP--) per nameservice.

No additional DataNode-specific properties are required beyond what is shown in Step 1. Restart DataNodes after deploying the updated configuration.

Step 3: Configure ViewFS Mount Table in core-site.xml

ViewFS configuration lives in core-site.xml (not a separate federation-site.xml) and must be deployed to all client nodes and all cluster nodes. The mount table maps directory prefixes to target nameservices.

<property>
  <name>fs.viewfs.mounttable.cluster.link./user</name>
  <value>hdfs://ns1/user</value>
  <description>Mount ns1's /user at /user in the viewfs://cluster/ namespace</description>
</property>
<property>
  <name>fs.viewfs.mounttable.cluster.link./data</name>
  <value>hdfs://ns2/data</value>
  <description>Mount ns2's /data at /data in the viewfs://cluster/ namespace</description>
</property>
<property>
  <name>fs.viewfs.mounttable.cluster.link./tmp</name>
  <value>hdfs://ns1/tmp</value>
</property>

<!-- Optional: default mount for unmatched paths (falls back to ns1) -->
<property>
  <name>fs.viewfs.mounttable.cluster.link./</name>
  <value>hdfs://ns1/</value>
</property>

Key points:

• The property name pattern is fs.viewfs.mounttable..link. where clusterName is arbitrary (here cluster) and path is the mount point in the unified view.
• The value is a fully qualified HDFS URI using the nameservice logical name (e.g., hdfs://ns1/user).
• Mounts are evaluated longest-prefix-first; order in the file does not matter.
• Clients access the unified namespace via viewfs://cluster/.

Step 4: Initialize and Start the Federation

1. Format each nameservice's NameNodes: On the first NameNode of each nameservice, run:

   hdfs namenode -format -clusterId CID- -force -nonInteractive

   Use the same clusterId for all NameNodes across all nameservices (this is required for federation). Then format the standby NameNodes with -bootstrapStandby.
2. Start JournalNodes: Start the JournalNode daemons on all JournalNode hosts (hdfs --daemon start journalnode).
3. Start NameNodes: Start the active and standby NameNodes for each nameservice.
4. Start DataNodes: Start DataNodes. They will register with all configured nameservices.
5. Create root directories: On the active NameNode of each nameservice, create the directories referenced in the ViewFS mount table:

   hdfs dfs -mkdir -p hdfs://ns1/user
   hdfs dfs -mkdir -p hdfs://ns1/tmp
   hdfs dfs -mkdir -p hdfs://ns2/data

Expected Checks and Diagnostics

• Federation configuration loaded: On any node, run:

  hdfs getconf -confKey dfs.nameservices

  Should return ns1,ns2.
• ViewFS mount table loaded: On a client node:

  hdfs getconf -confKey fs.viewfs.mounttable.cluster.link./user
  # Expected: hdfs://ns1/user

• DataNode block pools: Check DataNode logs for lines like:

  BPOfferService: Block pool BP-1234567890-10.0.0.1-12345 (namespace ns1) ...
  BPOfferService: Block pool BP-1234567890-10.0.0.1-12346 (namespace ns2) ...

  Two distinct block pool IDs confirm registration with both nameservices.
• Cluster report: Run:

  hdfs dfsadmin -report

  Output should show both nameservices with their respective NameNodes and live DataNodes.
• ViewFS routing test: On a client:

  hdfs dfs -ls viewfs://cluster/
  # Should show user/, data/, tmp/
  
  hdfs dfs -put /etc/passwd viewfs://cluster/user/test.txt
  hdfs dfs -put /etc/hosts viewfs://cluster/data/test.txt
  
  hdfs dfs -ls viewfs://cluster/user
  # File appears only here (routed to ns1)
  hdfs dfs -ls viewfs://cluster/data
  # File appears only here (routed to ns2)

• HA failover test: Trigger failover on one nameservice:

  hdfs haadmin -failover ns1 nn1 nn2

  Verify client operations continue without interruption.

Limitations and Operational Considerations

• Namespace scaling only: Federation does not increase data throughput linearly; it only scales namespace capacity and metadata operations. DataNode network and disk I/O remain shared resources.
• No cross-namespace moves: Moving a directory between nameservices (e.g., /user to /data) is not natively supported. You must copy data via distcp (which changes block pool IDs) and delete the source.
• HA complexity: Each nameservice requires its own QJM quorum. Plan for at least three JournalNodes per nameservice (they can be co-located but maintain separate edit log directories).
• Configuration consistency: Keep hdfs-site.xml and core-site.xml identical across all nodes. Use a configuration management tool (Ansible, Puppet, Cloudera Manager, Ambari) to enforce consistency.
• Single clusterId: All nameservices in a federation must share the same clusterId (set during initial NameNode format). Mismatched clusterIds will prevent DataNodes from registering correctly.

Recovery Options

If a NameNode fails to start after federation changes:

1. Check the NameNode log for clusterId mismatch errors; ensure all NameNodes were formatted with the same clusterId.
2. If ViewFS routing fails, validate core-site.xml syntax with hdfs getconf -confKey fs.viewfs.mounttable.cluster.link./user on a client node.
3. For DataNode registration issues, verify network reachability and that dfs.namenode.rpc-address.. properties match the active NameNode addresses.
4. Rollback: Revert hdfs-site.xml and core-site.xml to pre-federation versions and restart services. No metadata migration is required because namespaces remain independent.

Verification Checklist

Check	Command	Success Indicator
Nameservices defined	hdfs getconf -confKey dfs.nameservices	Returns ns1,ns2
ViewFS mount for /user	hdfs getconf -confKey fs.viewfs.mounttable.cluster.link./user	Returns hdfs://ns1/user
DataNode block pools	DataNode log grep Block pool	Two distinct block pool IDs (one per nameservice)
Client write to ns1	hdfs dfs -put /tmp/test viewfs://cluster/user/test	File appears in ns1 NameNode UI (port 9870)
Client write to ns2	hdfs dfs -put /tmp/test viewfs://cluster/data/test	File appears in ns2 NameNode UI (port 9870)
HA failover ns1	hdfs haadmin -failover ns1 nn1 nn2	Command succeeds; client ops continue

With these steps completed, your HDFS cluster can now scale its namespace horizontally, allowing you to add more nameservices as the number of files grows, while keeping the operational model simple for end users through ViewFS.