Keep remote long-running jobs alive with tmux detachable sessions

The problem and the takeaway

When you start a long-running command over SSH such as a data pipeline model training run or nightly build the moment your connection drops the process terminates and its output is lost. There is no native way to preserve the session state or resume viewing output after the fact unless you redirected output to a file before launching the command.

The useful takeaway is that tmux creates detachable terminal sessions: the process continues running on the server and you can reattach from any terminal even after a disconnect or reboot preserving both the running state and the scrollback history.

Prerequisites

• Tmux installed on the remote host. On most Linux distributions the package is named tmux; on macOS with Homebrew use brew install tmux.
• A shell account and the ability to SSH into the host.
• Familiarity with the default prefix key Ctrl+b. All tmux commands begin with this key followed by a secondary key.

Focused procedure

1. Start a named tmux session on the remote host:

tmux new -s work

The shell prompt changes and a status bar appears at the bottom of your terminal indicating you are inside a tmux session named work.

1. Launch your long-running command directly inside the session.

For example to keep a process alive while you step away:

sleep 3600

Replace sleep 3600 with your actual build training script or data migration command.

1. Detach the session without killing the process.

Press Ctrl+b release both keys then press d. Your terminal returns to the original shell but the tmux session and the sleep 3600 process continue running on the server.

1. (Optional) Exit SSH or close your local terminal.

The process keeps executing server-side. You can now start a new SSH connection from a different machine or close your laptop the job is unaffected.

Reattaching later

When you want to check progress or resume interaction:

1. Run tmux ls to list active sessions. You will see output similar to work 1 windows created Tue 00:12 120x40 tmux off.

tmux attach -t work

The terminal splits back into the tmux session showing the same prompt and any output generated since detachment. If you launched sleep 3600 the countdown continues from where it left off.

Expected checks

• After detaching and ending your SSH session reconnect to the host and run tmux ls. The session name you chose should appear.
• Run tmux attach -t work and verify the prompt is active. If you ran a long-running command confirm it is still progressing (e.g. the sleep countdown is still active or your programs output is scrolling).
• If you prefer not to reattach visually check the process from outside tmux: ps aux | grep your-command.

Recovery and durable logs

Tmux preserves in-memory session state and scrollback but the session is lost if the tmux server dies or the host reboots. For critical jobs pair tmux with a durable log file so you can review what ran even if the session disappears.

your-command 2>&1 | tee run.log

If you later need to inspect what executed, cat run.log is the source of truth. To clean up a stale session you no longer need, use it only when you are certain the session is no longer needed.

tmux kill-session -t work

Limitations and version notes

• Server reboot: tmux sessions do not survive a server reboot or an intentional tmux kill-server. For jobs that must survive restarts use a process supervisor such as systemd screen or a dedicated job scheduler alongside tmux.
• Key bindings and some command flags differ between tmux 2.x and 3.x. Check your version with tmux -V before relying on version-specific options such as history-limit adjustments.
• Scrollback history is bounded by the history-limit option. The default is a few thousand lines; very verbose output may be truncated. You can raise it interactively inside a session: press Ctrl+b release then press : type set history-limit 100000 and press Enter.
• Running tmux inside another terminal multiplexer or nested SSH sessions can cause prefix-key conflicts. If you use screen or another tmux instance set a distinct prefix (e.g. set -g prefix C-a) or detach the outer session first.

Quick verification checklist

1. On the target host run tmux -V and tmux ls to confirm installation and that no leftover sessions are active before you start work.
2. Perform an end-to-end test: start a session with tmux new -s work run sleep 300 detach with Ctrl+b d close the SSH connection reconnect and verify that tmux attach -t work shows the sleep process still running.
3. Check the effective scrollback limit with tmux show-options -g history-limit inside a session.
4. Simulate failure recovery by killing a test session with tmux kill-session -t work and confirming that any output you redirected with tee remains available as a file.