Nomad dev agent behavior limits regarding environment repeatability

Question

Ephemeral State and Configuration Drift
The nomad agent -dev command provides a single-process server and client bound to the loopback interface. While this simplifies local evaluation, the agent operates with an in-memory state. All jobs, allocations, and cluster configurations are discarded upon process restart, requiring full resubmission of workloads to restore the environment.
Furthermore, the dev agent applies permissive defaults that diverge from production-ready configurations. Specifically, the raw_exec driver is enabled by default and ACLs are disabled, which can mask permission errors or driver unavailability when moving jobs from a local development cycle to a secured cluster.
Repeatability Constraints
These behaviors create a gap between a convenient local sandbox and a repeatable development environment that mirrors staging or production. It remains unclear if the dev agent's ephemeral nature is sufficient for iterative testing or if parity requires a persisted, multi-node configuration using containers or VMs.

Does the lack of state persistence in dev mode preclude its use as a baseline for repeatable local workflows?
How can the gap between dev-mode driver defaults (like raw_exec) and production restrictions be reconciled without abandoning the -dev flag?

Tasadduq Burney · Answer

Repeatability in Nomad Dev Mode
The lack of state persistence in nomad agent -dev does not preclude its use for repeatable workflows, provided that the source of truth is shifted from the agent's memory to external version-controlled job specifications. Because the dev agent is ephemeral by design, repeatability is achieved through automation rather than persistence.

Reconciling Dev and Production Gaps
To maintain the convenience of the -dev flag while mitigating the risks of permissive defaults (like raw_exec and disabled ACLs), you must implement a validation layer that mimics production constraints. You cannot "toggle off" these defaults within the -dev flag itself, as they are hardcoded for rapid prototyping.

Implementation Steps for Environment Parity

Externalize Job State: Store all HCL job files in a git repository. Use a simple shell script or Makefile to re-submit all required workloads immediately after starting the dev agent.
Restrict Drivers via Configuration: While -dev enables raw_exec, you can override driver availability by providing a minimal configuration file. Run the agent with a config file that explicitly defines allowed drivers, even when using dev mode.
Simulate ACLs via CI: Since the dev agent ignores ACLs, integrate a "linting" or "dry-run" step in your CI/CD pipeline using nomad job plan against a staging cluster to catch permission errors before they reach production.
Avoid raw_exec for Logic: Use the docker driver locally. If the dev agent is running on a host with Docker, the docker driver will function, ensuring the isolation boundaries match production.

Verification Commands
To confirm the ephemeral nature and driver behavior, use these scoped commands:
# Verify state loss: Submit job, restart agent, then check status
nomad job run my-job.hcl
# (Restart nomad agent -dev process)
nomad job status my-job
# Verify ACL status in dev mode
nomad operator acl status

Diagnostic Requirement
To provide a more specific configuration override, please specify if you are running the dev agent on a native Linux host or within a containerized environment (e.g., Docker Desktop), as this changes how driver paths are mapped.

Nomad dev agent behavior limits regarding environment repeatability

Ephemeral State and Configuration Drift

Repeatability Constraints

1 answer

Repeatability in Nomad Dev Mode

Reconciling Dev and Production Gaps

Implementation Steps for Environment Parity

Verification Commands

Diagnostic Requirement

Your perspective belongs here.

0 question comments