Splunk Deployment Server ↔ Docker Splunk: Resolving Configuration Precedence Conflicts

Configuration Precedence: Docker vs. Deployment Server

p>When a Splunk instance running in Docker is managed by a Deployment Server (DS), the final value of a configuration setting is determined by Splunk's internal precedence hierarchy. However, Docker-specific environment variables and mounts introduce external layers that can complicate this.

In the specific conflict between a mounted app’s local/app.conf and a Docker-passed environment variable, the Docker environment variable typically takes precedence if it is used to write to a file in system/local/ or if it utilizes the SPLUNK_START_ARGS mechanism.

Splunk's standard precedence (from highest to lowest) is generally as follows:

• $SPLUNK_HOME/etc/local/ (System-level overrides)
• $SPLUNK_HOME/etc/system/local/ (Where Docker-generated configs often land)
• $SPLUNK_HOME/etc/apps//local/ (Deployment Server pushed configs)
• $SPLUNK_HOME/etc/apps//default/ (App defaults)
• $SPLUNK_HOME/etc/system/default/

If your Docker setup uses environment variables to inject settings into a file located in system/local/, those will override any settings pushed by the Deployment Server to the app's local/ directory. To ensure the Deployment Server maintains control, avoid using Docker environment variables to write to higher-precedence directories than the apps//local path.

The Role of the Phonehome Interval

The Deployment Server phonehome interval (controlled by refresh.interval in server.class.conf) determines how often the client checks for new updates. In a CI/CD pipeline, this creates a latency gap between the creation of a container image and the application of new configurations.

• Default Behavior: By default, the interval is often set to 6000 seconds (1 hour). A new container instance may wait up to this duration before pulling the latest app bundle.
• Impact on CI: Shortening this interval reduces the delay but increases the load on the DS, especially if many containers spin up simultaneously.
• Resolution: For deterministic builds, do not rely on the background interval. Instead, trigger a manual sync using the splunk pull-server command within your container's entrypoint script or a startup task to ensure the bundle is present immediately before the service accepts traffic.

Verification Steps

To verify which source is currently in effect, execute the following command within the running container:

splunk show config -f  --value

The output will explicitly state the file path where the value is being pulled. If the path points to system/local, the Docker-level override is winning; if it points to apps//local, the Deployment Server is in control.