High‑Availability Workloads in Nomad with Multi‑Region Federation

Problem: Single‑Region Failures Keep Your Service Down

Many teams deploy Nomad jobs in a single datacenter or cloud region. If that region loses connectivity, the entire job set goes offline. Even with Consul health checks, the client still tries to hit the unhealthy nodes, causing outages for end users. The question is: how can we keep a Nomad job running when an entire region becomes unreachable?

Thesis: Use Nomad’s Multi‑Region Federation to Auto‑Failover

Nomad’s multi‑region federation lets you run a single job specification across geographically separated clusters. When one region fails, the remaining regions automatically continue to serve the job, and Nomad clients dial the healthy region without manual intervention. This feature is built on top of Consul’s service mesh and requires only a declarative configuration block.

How Federation Works Under the Hood

1. Shared Consul Cluster: Each Nomad region registers with the same Consul cluster. The Consul agents act as a global service registry and health‑check aggregator.

2. Federation Block: In the nomad.hcl server configuration, add a federation stanza listing all region names. Example:

server {
  region = "us-east"
  bootstrap_expect = 3

  federation {
    region = "us-west"
    region = "eu-central"
  }
}

3. Automatic Allocation Spread: Once federation is enabled, the scheduler sees all regions as peers. It balances allocations across them based on resource availability.

4. Client Dial‑In: Nomad clients query the local Consul agent for healthy nodes. If the local region is down, the Consul agent still reports nodes from the healthy region, so clients automatically route to them.

Concrete Example: Deploy a Consul‑Connected HTTP Service Across Two Regions

Assumptions: Nomad 1.4+, Consul 1.13+, three servers per region.

1. Set Up the Consul Cluster
   Run a single Consul cluster that all regions can reach (e.g., a highly‑available Consul cluster in a neutral VPC). Ensure each region’s Nomad servers have the Consul client address configured.
2. Configure Nomad Servers
   Place the federation block in nomad.hcl for both regions, using the same Consul cluster address.
3. Submit the Job
   Use a simple HCL job that registers with Consul:

   job "http-service" {
     datacenters = ["us-east", "us-west"]
   
     group "web" {
       task "server" {
         driver = "docker"
         config {
           image = "hashicorp/http-echo:latest"
           args  = ["-text=Hello from Nomad"]
         }
   
         service {
           name = "http-service"
           port = "http"
           tags = ["env:prod"]
           check {
             name     = "http"
             type     = "http"
             path     = "/"
             interval = "10s"
             timeout  = "2s"
           }
         }
   
         resources {
           network {
             mbits = 10
             port "http" {}
           }
         }
       }
     }
   }
   

   Run nomad run http-service.hcl on either region. The job appears in both.

4. Verify Allocation Visibility
   From the us-east region, run nomad status -region=us-east http-service and nomad status -region=us-west http-service. Both should show allocations in their respective regions.
5. Simulate a Region Outage
   Stop the us-west Nomad servers. Use nomad status http-service from a client in us-east. The job remains healthy, and requests to http-service are served by the us-east region.

Trade‑Offs and Limitations

• Latency & Bandwidth: Cross‑region health checks and allocation coordination add network latency. Large job allocations or frequent health checks can increase bandwidth usage.
• Uneven Resources: If regions have different capacities, the scheduler may spread allocations unevenly, potentially overloading a smaller region.
• Stateful Workloads: Federation is best suited for stateless services. Stateful services require external data replication (e.g., databases, object stores) because Nomad does not replicate state across regions.
• Consul Dependency: All regions must reach the same Consul cluster. Network partitions that isolate Consul break federation.
• Version Compatibility: All regions must run the same major Nomad version (1.4+). Mixing older versions breaks federation.

Actionable Next Steps

1. Set up a shared Consul cluster or use Consul Cloud to avoid managing a separate cluster.
2. Add the federation stanza to each region’s nomad.hcl and restart servers.
3. Deploy a test job and confirm allocations appear in all regions.
4. Monitor Consul and Nomad logs for federation events to ensure automatic failover is working.
5. Measure cross‑region request latency and adjust resource reservations or network placement accordingly.
6. For stateful services, integrate external replication mechanisms (e.g., CockroachDB, etcd, S3) and ensure idempotent job restarts.

By following these steps, you can turn Nomad’s multi‑region federation into a robust high‑availability strategy that automatically keeps your services online, even when an entire region goes down.