Building a Secure Multi‑Tenant Nomad Cluster with Namespaces, ACLs, and Node Pools

Problem Statement

Organizations that run multiple teams or customers on a single Nomad cluster must guarantee that workloads cannot interfere with one another. The goal is to isolate tenants while keeping operational overhead low and avoiding the need for separate clusters.

Key Requirements

• Workload isolation: Jobs from tenant A must never run on a node intended for tenant B.
• Predictable scheduling: Batch and service jobs must be scheduled according to per‑tenant quotas and resource limits.
• Auditable access control: All API operations must be logged and enforce least‑privilege ACL tokens.
• Minimal operational overhead: No separate Nomad cluster per tenant; the design should fit into a single HA server set.

Minimal Viable Architecture

Assuming Nomad 1.4+ (TLS, namespaces, node pools, ACLs, and CSI support), the smallest cluster that satisfies the requirements is:

• Three Nomad servers (HA) with TLS enabled.
• Client nodes grouped into node pools by role or sensitivity.
• One namespace per tenant (e.g., tenant‑alpha, tenant‑beta).
• ACL policies that grant create, read, update only within the tenant’s namespace and node pool.
• External secrets integration (Vault, etc.) to keep sensitive data off the Nomad job spec.

Cluster Layout

Nomad Server Set (3 nodes)
├── TLS certs & shared key
├── ACL master token (root)
└── Namespace and ACL policy store

Client Nodes
├── pool‑alpha (tenant‑alpha workloads)
├── pool‑beta (tenant‑beta workloads)
└── pool‑shared (public / shared services)

Namespace and ACL Policy Example

Below is a minimal policy for tenant‑alpha. Replace tenant-alpha-token with the actual token when creating the ACL token for the tenant.

# tenant-alpha.hcl
namespace "tenant-alpha" {
  policy = "write"
}

acl "tenant-alpha-policy" {
  namespace = "tenant-alpha"
  capabilities = ["create", "read", "update", "delete"]
  rules = <

To create the token:

# Run on a Nomad server
nomad acl token create -policy=tenant-alpha-policy -description="Tenant Alpha token"

Node Pool Definition

Node pools are defined in the client configuration using the node_class setting. All nodes in the same pool share the same class name.

# /etc/nomad.d/client.hcl (tenant‑alpha node)
node_class = "pool‑alpha"

Job Constraint to Enforce Pool and Namespace

Each job must include constraints that bind it to the correct pool and namespace. The following example shows a batch job for tenant‑alpha.

job "alpha‑batch" {
  namespace = "tenant-alpha"
  type = "batch"

  group "default" {
    task "run" {
      driver = "exec"
      config {
        command = "/usr/bin/python"
        args    = ["-c", "print('Hello from tenant alpha')"]
      }
    }
  }

  constraint {
    attribute = "${node.class}"
    value     = "pool-alpha"
  }
}

Trust and Data Boundaries

• Control Plane Trust: Nomad servers are trusted; they hold ACL master tokens and store namespace definitions. Clients authenticate via TLS and present ACL tokens.
• Node Pool Constraints: Logical isolation—jobs are prevented from scheduling on the wrong pool by the constraint. However, workloads on the same host still share the kernel; additional OS‑level controls (e.g., cgroups, SELinux) are required for stronger isolation.
• Secrets Handling: Job specs should reference Vault secrets via the vault stanza. The secret values are injected at runtime and never stored in the Nomad state.
• Network Segmentation: Use host‑level firewall rules or CNI plugins to restrict inter‑pool traffic.

Operational Checks

1. Server Quorum – nomad server members should show all servers in Leader or Follower state. A missing member triggers a scheduling freeze.
2. Client Heartbeat – nomad node status must list every client as ready. Clients in draining state should be re‑deployed or removed.
3. Allocation Placement – nomad alloc list -namespace=tenant-alpha should show only allocations with node_class=pool-alpha. Any allocation outside the pool indicates a misconfiguration.
4. ACL Token Expiry – nomad acl token list shows ExpiresAt. Tokens should be short‑lived (e.g., 24h) and rotated via automation.
5. Namespace Quota – Nomad now supports quota on namespaces. Verify via nomad namespace info tenant-alpha that resource limits are enforced.
6. CSI Volume Attachment – For stateful jobs, run nomad job status -verbose <job-id> and inspect Volumes to ensure they attach to the correct node pool.

Failure Modes and Design Change Triggers

• Server Quorum Loss: If the server set loses quorum, the cluster stops scheduling. Solution: add a fourth server or enable a fallback leader election mechanism.
• Client Node Failure: A node goes offline; allocations are rescheduled to other nodes. If the node was the sole member of a pool, the job may be unschedulable. Trigger: add redundancy to critical pools.
• Namespace or Node Pool Misconfiguration: A job is scheduled in the wrong pool or namespace. Trigger: implement automated policy checks in CI or use Nomad’s policy check API.
• ACL Token Leakage: A token is exposed. Immediate revocation via nomad acl token revoke and rotation of the master token.
• Secret Exposure: Secrets leaked via job specs. Trigger: enforce policy that disallows inline secrets and audit job specs for vault stanzas only.

When to Re‑Design

Consider changing the architecture if:

• Tenant workloads grow beyond the capacity of a single node pool and require dedicated hardware.
• Isolation requirements evolve to enforce kernel‑level isolation (cgroups, seccomp).
• The number of tenants exceeds the limits of the ACL policy engine or the namespace quota system.
• Operational complexity (e.g., manual token rotation) outweighs the benefits of a single cluster.

Practical Verification Checklist

1. Run nomad namespace list and confirm each tenant namespace exists.
2. Execute nomad acl policy list and verify policies are scoped correctly.
3. Use nomad node status to ensure each client node reports the expected node_class.
4. Deploy a test job with the constraint and namespace, then run nomad alloc status to confirm placement.
5. Review the Nomad server logs for any ACL denied or constraint mismatch messages.

Limitations

• Namespace isolation is logical; workloads on the same host share the kernel. Use additional OS controls for stronger isolation.
• ACL tokens are bearer tokens; secure storage and rotation are the responsibility of the operator.
• Node pool constraints rely on correct labeling; mislabeling can expose tenants to each other.
• Secret injection via Vault requires the Vault server to be highly available; otherwise jobs may fail to start.