Zero‑Trust Service Mesh with Nomad and Consul Connect: A Minimal Architecture Guide

Problem Statement

Deploying microservices on Nomad often requires secure, mutual‑TLS communication without manually rotating certificates or exposing services to the host network. Consul Connect provides a sidecar injection that satisfies these needs, but the minimal configuration that delivers a zero‑trust mesh in a single data center is not always obvious. This guide shows the smallest suitable design, the trust boundaries it establishes, the operational checks you should run, and the failure modes you must guard against.

Design Requirements

• Nomad 1.4+ (or the latest stable release) with the connect stanza enabled.
• Consul 1.15+ with ACLs turned on and a single data‑center cluster.
• Nomad servers and clients running on the same hosts as Consul servers (or with network connectivity to them).
• TLS mutual authentication automatically handled by Consul Connect; no external certificate authority required.
• Sidecar injection per job, no external service mesh.

Minimal Architecture

Component	Role
Nomad Server	Job scheduler, state store, orchestrator.
Nomad Client	Runs application and sidecar containers.
Consul Server Cluster	Service catalog, key/value store, Connect mesh.
Consul Connect Sidecar	Establishes mutual TLS tunnels between services.
Application Service	Business logic, exposed only internally.
Client Network	External traffic, not reachable by services.

All inter‑service traffic is routed through the Consul Connect sidecar, ensuring encryption and authentication. The application never listens on the host network; it only exposes a local port that the sidecar forwards to the Consul agent.

Trust and Data Boundaries

The sidecar defines a clear boundary:

• Inbound to the service: Only the sidecar can reach the application port. The host network is blocked.
• Outbound from the service: All traffic must pass through the sidecar, which encrypts it and authenticates the destination via Consul Connect.
• Sidecar to Consul: Mutual TLS certificates are issued by Consul, scoped to the service name. No shared keys are stored on disk.

ACL tokens are scoped per job, limiting what the sidecar can query from Consul. Misconfigured ACLs can break service discovery or tunnel establishment, so ACL policies must be reviewed before deployment.

Nomad Job Example

Below is a minimal job file that demonstrates sidecar injection. Replace placeholders with your environment values.

job "webapp" {
  datacenters = ["dc1"]
  type        = "service"

  group "app" {
    task "web" {
      driver = "docker"
      config {
        image = "myorg/webapp:latest"
        port_map {
          http = 80
        }
      }
      resources {
        network {
          mbits = 10
          port "http" {}
        }
      }
      service {
        name = "webapp"
        port = "http"
        tags = ["urlprefix-/app"]
        connect {
          sidecar_service {}
        }
      }
    }
  }
}

Key points:

• The connect stanza inside the service block triggers sidecar injection.
• No TLS configuration is required in the job file; Consul Connect supplies certificates automatically.
• The sidecar will listen on localhost:8080 (or the port you configure) and forward traffic to the Consul agent.

Operational Checks

1. Verify sidecar presence
   

   ssh nomad-client-01
   sudo docker ps | grep connect
   # Expected: a container named "consul-connect-webapp-..."
   

2. Confirm mesh health
   

   ssh nomad-client-01
   consul connect status
   # Expected: "Mesh health: healthy" and list of active tunnels
   

3. Check service registration
   

   curl http://localhost:8500/v1/catalog/service/webapp
   # Expected: JSON list containing the service instance
   

4. Validate TLS termination
   

   curl -k https://127.0.0.1:8080/api/health
   # Expected: 200 OK
   

5. Monitor certificate rotation
   

   watch consul kv get -format=json "consul/connect/webapp/ca"
   # Certificates should be refreshed automatically; no manual action needed.
   

Failure Modes & Mitigations

Failure	Effect	Mitigation
Sidecar crash	Service becomes unreachable; Consul health check fails.	Nomad’s restart_policy restarts the task; Consul auto‑reestablishes the tunnel.
Certificate expiration	TLS handshakes fail; services cannot connect.	Consul renews certificates automatically; monitor consul connect status for renewal events.
Network partition between Nomad client and Consul server	Sidecar cannot register; services are isolated.	Consul’s health checks mark the service as critical; Nomad can be configured to pause the job.
ACL misconfiguration	Sidecar cannot discover other services; traffic fails.	Validate ACL policies with consul acl policy list and consul acl token list.

Conditions That Would Change the Design

• Multi‑data‑center deployment – Requires Consul WAN federation or replication. Sidecar injection remains, but you must ensure the Consul cluster spans datacenters and that ACLs are replicated.
• High‑throughput, low‑latency workloads – Sidecar adds a small overhead. Benchmark latency; if unacceptable, consider deploying the service without a sidecar and using explicit TLS.
• External service mesh integration – If you need to expose services to the public internet, you’ll need an ingress gateway (e.g., Consul Connect Gateway) in addition to the sidecar.
• Dynamic port allocation – For jobs that require random ports, ensure the port_map and service.port are correctly aligned.

Summary

By combining Nomad’s connect stanza with a Consul server cluster, you can achieve a zero‑trust service mesh in a single data center with minimal infrastructure. The sidecar automatically handles TLS, certificate rotation, and service discovery, keeping your application code simple. Operational checks such as consul connect status and Nomad health checks give you confidence that the mesh remains healthy. When you move to multi‑region or latency‑sensitive workloads, revisit the design to add WAN federation or consider alternative TLS strategies.