Choosing Nomad Native Service Discovery Over Consul for East‑West Traffic in a Nomad Cluster

Problem and Takeaway

When a Nomad cluster needs to locate services for east‑west traffic, the decision is whether to rely on Nomad’s built‑in service discovery (available since 1.3) or to continue using Consul for service registration and lookup. The practical takeaway is that native discovery can replace Consul for simple, unencrypted east‑west calls, but Consul remains necessary when encrypted traffic, multi‑cluster mesh, or a robust DNS ecosystem is required.

Requirements

• Nomad version 1.3 or newer – native service discovery was introduced in this release.
• Access to the Nomad server’s HTTP API or client driver privileges to register services.
• Network connectivity between Nomad clients and the built‑in DNS server (typically localhost:8127).
• Optional: Consul cluster if mTLS or cross‑cluster service mesh is needed.

Minimal Suitable Design

For a cluster that only needs unencrypted east‑west communication, the smallest design consists of:

1. Jobs that declare a service block with service {} and a provider of type nomad.
2. The built‑in DNS server automatically resolves service.service to the IP and port of each allocation.
3. Job templates that render the upstream address using the NOMAD_SERVICE_NAME and NOMAD_SERVICE_PORT environment variables.

This design avoids any external dependency on Consul and requires only the default ACL and TLS configuration of the Nomad cluster.

Trust Boundaries

Native discovery data lives inside the Nomad state, which is protected by the cluster’s ACLs and TLS. It therefore inherits the same trust model as scheduling decisions: only principals authorized to register or modify services can affect the discovery table. Unlike Consul Connect, native discovery does not provide SPIFFE‑based identity or encrypted traffic; it only offers name‑to‑address mapping.

Operational Checks

• Query the DNS interface directly: dig @127.0.0.1 -p 8127 <service>.service. Successful resolution confirms that the client can reach the built‑in DNS server.
• Observe template re‑rendering on service change events. After a nomad job run updates a service registration, the rendered address in a dependent job should change within the client’s sync interval (default 10 seconds).
• Monitor client‑side check failures. Native health checks are executed by the client; a failing check will mark the allocation as dead and trigger rescheduling. Alert on a rise in client check failures rather than assuming anti‑entropy will heal registrations.

Failure Modes

If the Nomad server loses quorum, the state that stores service registrations becomes temporarily unavailable. Clients continue to serve cached DNS entries, but new registrations and updates may be delayed until quorum is restored, potentially causing brief outages for newly started allocations.

In contrast, a separate Consul cluster can continue serving stale but available service records during a Nomad control‑plane outage, providing higher availability for discovery queries at the cost of eventual consistency.

When the Design Changes

• Zero‑trust networking or required mTLS for east‑west traffic pushes the decision toward Consul Connect, which supplies SPIFFE identities and encrypted service calls.
• Non‑Nomad clients (e.g., external micro‑services) that must resolve service names without a Nomad client driver also benefit from Consul’s universal DNS endpoint.
• Federated clusters across regions may require a dedicated service registry to avoid cross‑region latency and to maintain consistent discovery semantics.

Concrete Example and Verification

To verify that native discovery works on your target Nomad version, deploy a minimal two‑job test cluster:

1. On a Nomad client with client driver privileges, create a job definition that registers a service named demo with a service {} block and provider nomad. Example snippet:

job "demo" {
  datacenters = ["dc"]
  type = "service"
  service {
    name = "demo"
    port = 8080
    provider = "nomad"
  }
}

2. Run the job: nomad job run demo.hcl. The client registers the allocation and updates the built‑in DNS entry.
3. From any client, query the DNS name: dig @127.0.0.1 -p 8127 demo.service. Verify that the response contains the IP and port of the running allocation.
4. Modify the service port in the job file, re‑run nomad job run, and observe the DNS entry update after the client’s sync interval. Time the convergence to confirm it meets your SLO.
5. Simulate a quorum loss by stopping a majority of Nomad servers. Continue to issue DNS queries; note that responses may become stale or time out, documenting the degradation for your runbook.

These steps provide a practical way to confirm that native discovery behaves as expected, that template rendering converges within acceptable latency, and that the system degrades gracefully under server failure.

Conclusion

Native Nomad service discovery offers a lightweight, dependency‑free solution for east‑west traffic when encryption and advanced mesh features are not required. Its minimal design reduces operational overhead, but it inherits the cluster’s trust model and has limitations in health‑check semantics and DNS performance. Consul‑integrated discovery remains justified when zero‑trust networking, multi‑cluster federation, or a mature DNS ecosystem is needed. By validating with a controlled test cluster and monitoring the outlined operational checks, teams can confidently choose the appropriate discovery mechanism for their Nomad environment.