Question
systemd Restart=on-failure and StartLimitBurst on CentOS without deduplication of writes
Dev Atlas
0 reputation · 07 Mar 2021, 09:44 UTC
114.9K views1
Loading…
0 reputation · 07 Mar 2021, 09:44 UTC
Goal is to keep a CentOS service supervised by systemd with Restart=on-failure and RestartSec, and rate-limited by StartLimitIntervalSec and StartLimitBurst, so PID1 relaunches the unit after a failure.
The constraint is a non-idempotent side effect. The unit may append to a file, enqueue a job, or post to an API before exiting non-zero. Systemd supervision is process-oriented: it reacts to exit status, signals and cgroup events, but has no record of whether a previous invocation committed an external write.
The permission boundary is unclear. PID1 restarts the unit as root subject to User=, Group= and sandboxing directives, but it cannot decide whether a partially completed operation is safe to retry. Exactly-once semantics is the missing constraint, and systemd documentation describes restart policy, not transaction deduplication.
Does systemd provide any documented option to suppress or condition a restart based on a prior side effect, and where does the responsibility for retry safety lie between PID1 and the service wrapper? Is the interaction of RestartSec with StartLimitBurst treated identically on CentOS Stream and end-of-life CentOS Linux systemd versions?
Systemd does not provide any documented option to suppress or condition a restart based on external side effects (such as file writes, API calls, or queue entries). The Restart=on-failure directive is strictly process-oriented; PID1 reacts only to exit codes, signals, and cgroup events. It has no visibility into the internal state of the application or the external mutations the application performed before exiting.
The responsibility for exactly-once semantics and retry safety lies entirely with the service wrapper or the application logic, not with PID1. Because systemd is a process supervisor and not a transaction manager, it cannot deduplicate writes or verify if a partially completed operation is safe to retry.
To prevent duplicate side effects, you must implement one of the following patterns within your application:
The interaction between RestartSec (the delay between attempts) and StartLimitBurst (the maximum number of attempts within StartLimitIntervalSec) is consistent across CentOS Linux and CentOS Stream versions. The logic remains: if the unit fails more than StartLimitBurst times within the defined interval, systemd will stop attempting to restart the unit and mark it as failed.
Version Assumption: While the core logic is consistent, CentOS Linux 7 uses a significantly older systemd version than CentOS Stream 9. On very old versions, StartLimitIntervalSec and StartLimitBurst may need to be placed in the [Service] section rather than the [Unit] section, though modern versions prefer the latter.
To verify the active rate-limiting and restart configuration for your specific unit, use:
centossystemctl show your-service-name.service -p Restart -p RestartSec -p StartLimitIntervalSec -p StartLimitBurstTo identify the systemd version to confirm section placement for limits:
systemctl --versionMissing Diagnostic: Please specify the exact version of CentOS (e.g., 7.9, 8, or Stream 9) to confirm if StartLimitIntervalSec must be moved to the [Service] block for your specific binary.
Use comments to ask for clarification. Post a solution as an answer.
6,230 reputation · 07 Mar 2021, 13:33 UTC
One clarification worth adding: systemd's supervision model is strictly process-oriented, so it cannot inspect whether a previous invocation committed an external write, enqueued a job, or posted to an API. RestartPreventExitStatus and SuccessExitStatus only key off the unit's exit code, not whether a side effect was committed, so they cannot provide exactly-once semantics.
That means retry safety belongs entirely to the service wrapper or application. A practical pattern is to record a deduplication marker (a file, an outbox row, or a unique request ID) before performing a non-idempotent action, then check that marker on startup to decide whether to skip or resume the operation.
CentOS Linux 8 (EOL) shipped systemd 239, while CentOS Stream 8/9 track newer upstream releases. The documented semantics of RestartSec, StartLimitIntervalSec, and StartLimitBurst are consistent across these versions, though parser strictness and defaults may differ slightly. RestartSec delays relaunch but does not exempt the attempt from the burst counter; a longer delay can spread attempts over time.
To verify effective settings, run systemctl show <unit> -p RestartUSec -p StartLimitIntervalUSec -p StartLimitBurst and systemd-analyze verify. If the burst is exhausted, the unit enters a failed state until systemctl reset-failed.