Nomad and Consul Connect resource overhead for low-traffic workloads

For low-traffic workloads, the decision to use Consul Connect involves a trade-off between a persistent resource baseline and automated security. While native Nomad service discovery eliminates the sidecar, it removes the managed security layer provided by the service mesh.

Baseline Resource Overhead

An Envoy sidecar introduces a fixed resource floor regardless of request volume. In a typical Nomad allocation, a minimal Envoy proxy generally consumes between 20MB and 100MB of RAM and a small slice of CPU to maintain its connection to the Consul control plane and process routing table updates.

In a cluster with many small services, this "sidecar tax" can exceed the resource consumption of the actual application logic, leading to inefficient bin-packing and increased infrastructure costs.

Security Trade-offs: Connect vs. Native Discovery

Switching from Consul Connect to native Nomad service discovery (provider = "nomad") results in the loss of several critical security and traffic features:

• Automatic mTLS: You lose the automatic encryption of traffic in transit. Native discovery provides the IP address of the target service, but the communication is plaintext unless the application handles TLS manually.
• Identity-based Authorization: Consul "Intentions" (which define which services are allowed to talk to each other) are no longer enforced, as there is no proxy to intercept and validate requests.
• Observability: You lose the standardized L7 metrics (request rates, error codes, and latency) provided by the Envoy proxy.

Tuning Sidecar Resources

You can tune the resources block for the sidecar in your Nomad job specification to minimize costs. However, there is a stability threshold:

1. Memory Limits: Setting memory limits too aggressively (e.g., below 30-50MB) risks OOM (Out of Memory) kills during Consul catalog updates or when the proxy reloads its configuration.
2. CPU Limits: CPU can be tuned more loosely, as Envoy is highly efficient during low traffic, but it still requires bursts of cycles during startup and configuration synchronization.

Verification Steps

To determine the actual impact on your specific workload, use the following scoped commands:

# Compare memory usage across allocations
nomad alloc status <alloc-id>

To verify the loss of security when moving to native discovery, you can use tcpdump on a node to confirm that traffic between services is no longer encrypted.

Diagnostic Detail Needed: Are you currently using Consul Intentions to enforce zero-trust networking, or is the mesh primarily used for mTLS encryption?