Question
OpenSSH transport failure and remote command idempotency
Neel Byte
0 reputation · 23 Sept 2024, 13:47 UTC
64.8K views
Loading…
0 reputation · 23 Sept 2024, 13:47 UTC
When executing a non-idempotent command via OpenSSH, a network disconnection during the write phase can leave the remote process in an ambiguous state. Because the SSH protocol does not provide a built-in mechanism to track the completion status of a remote command after a transport-layer failure, the client cannot natively determine if the command was fully executed, partially executed, or never started.
While ServerAliveInterval and ServerAliveCountMax manage timeout detection, they do not facilitate the recovery of a session or the deduplication of writes. This creates a risk where external scripts wrapping SSH calls in retry loops may inadvertently trigger duplicate writes to the remote filesystem.
ControlMaster and ControlPersist alter the server-side process persistence when the primary transport connection is severed?33275 reputation · 23 Sept 2024, 15:29 UTC
No on both counts. ControlMaster and ControlPersist do not preserve server-side processes when the primary transport dies; they only multiplex additional sessions over one TCP connection. When that connection breaks, every multiplexed channel is closed and the remote processes attached to them are terminated. There is also no documented OpenSSH client option to query the exit status of a session that ended in a transport failure. Once the channel is lost, the client has no protocol handle to ask the server what happened.
While the client is writing command data, the remote sshd may have already spawned the shell, the shell may have started the command, or the command may have finished its side effects but lost the exit-status reply. The SSH transport protocol provides no post-hoc status query for a dead channel. ServerAliveInterval and ServerAliveCountMax only control how quickly the client detects a dead peer; they do not resume the channel, replay writes, or deduplicate commands.
Do not retry a non-idempotent command blindly. Make the remote action self-verifying with a completion marker:
ssh#!/bin/sh
MARKER=/var/tmp/job.done
[ -f "$MARKER" ] && exit 0
if important_command; then
touch "$MARKER"
fi
Then gate the retry on the marker:
ssh host 'test -f /var/tmp/job.done || /path/to/wrapper.sh'If the marker is absent after a reconnect, the work either never started or did not finish; the wrapper can safely be run again because the marker gates the side effects. For extra safety, have the remote command write to a temporary path and atomically rename into place, so a partial write is never mistaken for success.
This assumes a standard OpenSSH client/server (recent 8.x/9.x) and a POSIX remote shell, and that the remote filesystem persists across reconnects. If the remote action cannot be wrapped—for example, an interactive binary or a third-party tool with no completion signal—then no client-side retry loop is safe without an external coordinator such as a database transaction or a remote supervisor.
Use comments to ask for clarification. Post a solution as an answer.
33,275 reputation · 23 Sept 2024, 17:53 UTC
To build on the previous answer, it is important to distinguish between a transport-layer failure (TCP timeout/disconnect) and a session termination. While the SSH client loses the handle to the process, the remote process does not always terminate immediately upon a network drop.
Depending on the remote shell's configuration and how the command handles SIGHUP, a process may continue running as an orphan on the server even after the client detects a timeout. This increases the risk of "ghost" executions where a retry loop starts a second instance of a non-idempotent task while the first is still silently processing.
To verify this behavior in your environment (assuming OpenSSH 8.x/9.x), you can test the following:
sleep 300) via SSH.kill -9 the local SSH process or disconnect the network).ps aux | grep sleep to see if the process persisted.If processes persist, application-level locking (like flock) is required in addition to the completion markers mentioned previously.