HDFS Federation ViewFs mount table synchronization across distributed clients

Answer to the Question

Supported methods for automating ViewFs mount‑table synchronization:

• Periodic client‑side refresh – set dfs.viewfs.refresh.interval (seconds) in hdfs-site.xml. The client reloads the table automatically at the interval.
• On‑demand refresh command – run hdfs viewfs -refreshMountTable on every client (or via a script). The command forces an immediate reload.
• Cluster‑wide push – execute hdfs dfsadmin -refreshViewFs on a ViewFs NameNode. The command sends a refresh request to all registered ViewFs clients.
• Central configuration distribution – store the mount table in a shared location (HDFS file, ZooKeeper node, or a configuration‑management system). Clients read the file at startup or at the configured interval.
• Configuration‑management automation – use Ansible, Chef, Puppet, or similar to push updated hdfs-site.xml to all nodes and trigger the refresh command or restart the client processes.

What happens if a client uses an outdated mount table after a namespace migration?

• The client will route the request to the old NameNode based on the stale mapping.
• Typical outcomes are FileNotFoundException or NameNodeNotFoundException because the target NameNode no longer hosts the requested namespace.
• ViewFs does not automatically detect or correct the mismatch; the client must refresh its table or be restarted.

Practical Steps for a Large Fleet

1. Decide a refresh cadence. For most deployments, dfs.viewfs.refresh.interval=300 (5 min) balances freshness with traffic.
2. Centralize the mount table. Store the latest viewfs-mount-table.xml in HDFS under /etc/hadoop/viewfs/mount-table.xml or in ZooKeeper at /hadoop/viewfs/mount-table.
3. Configure clients to read the central source.

   dfs.viewfs.mount.table.file=/etc/hadoop/viewfs/mount-table.xml
   dfs.viewfs.refresh.interval=300

4. Deploy the new file with a configuration‑management tool.

   # Example Ansible task
   - name: Deploy ViewFs mount table
     copy:
       src: mount-table.xml
       dest: /etc/hadoop/viewfs/mount-table.xml
       owner: hdfs
       group: hadoop
       mode: '0644'
     notify: refresh viewfs
   
   - name: Refresh ViewFs on clients
     command: hdfs viewfs -refreshMountTable
     when: inventory_hostname in groups['hdfs_clients']

5. After a namespace migration, force an immediate refresh.

   # On a ViewFs NameNode
   hdfs dfsadmin -refreshViewFs
   # Or run the command on all clients via your automation tool
   hdfs viewfs -refreshMountTable
   

6. Verify the refresh.

   # On a client
   hdfs viewfs -printMountTable | grep -i /new-namespace
   # Check client logs for "ViewFs mount table refreshed"
   

Verification Checklist

• Run hdfs viewfs -printMountTable before and after a change to confirm the table updates.
• Check client logs for the message ViewFs mount table refreshed after a manual or interval‑based refresh.
• Attempt to access a path that was moved during migration; a 404 or NameNodeNotFoundException indicates the client still uses the old table.
• After issuing hdfs dfsadmin -refreshViewFs, ensure all clients log a refresh event within seconds.

Assumptions & Uncertainties

These instructions assume a Hadoop 3.x cluster with ViewFs enabled. If you are on an older Hadoop release, the hdfs viewfs command may not exist, and a client restart will be required to pick up a new mount table.

In a highly dynamic federation, consider reducing dfs.viewfs.refresh.interval further, but be aware that very short intervals increase metadata traffic.

Question for Further Clarification

Could you confirm the Hadoop version you are running? It influences whether the hdfs viewfs command is available and whether the dfsadmin -refreshViewFs option is supported.