Orphaned compiler processes persist after Ctrl+C in parallel Spack builds

Intended SIGINT behavior

When a user presses Ctrl+C during a spack install -j N command, Spack installs a SIGINT handler that:

1. Calls os.killpg with SIGINT for the process group of each tracked child job.
2. Waits for all those child PIDs to exit.
3. Runs a trap that removes the $SPACK_STAGE directory.

If every child respects SIGINT and terminates, the stage directory is cleaned up automatically.

Confirmed fact (Spack ≤0.20)

The handler indeed uses os.killpg with SIGINT. However, if a child process:

• Creates a new process group (e.g., via setpgid or double‑fork),
• Installs its own SIGINT handler that ignores or blocks the signal,
• Spawns grandchildren that are not in the original process group,

then os.killpg does not reach those processes, they can survive as orphans, and the cleanup trap never runs because Spack is still waiting for the tracked PIDs.

Likely explanation for leftover $SPACK_STAGE

Cleanup only proceeds after all tracked child PIDs have exited. Lingering compiler or make processes block the trap, leaving the stage directory locked. A timeout‑induced SIGALRM may kill the Spack wrapper but not the compiler, producing the same effect.

Steps to verify the current behavior

1. Start a parallel build: spack install -j 4 <pkg>.
2. Press Ctrl+C.
3. Immediately run:

   ps -ef | grep -E '(cc|make|ld|as)'

   to see if any compiler/make remnants remain.
4. Check whether $SPACK_STAGE for that build still exists:

   ls -d $SPACK_STAGE/*

   .

If processes remain or the stage directory persists, note the PID, PPID, PGID, and full command line of the lingering process (e.g., via ps -o pid,ppid,pgid,command -p <pid>). That detail tells us whether the process changed its process group or is ignoring SIGINT, which determines the next mitigation (e.g., using a wrapper that forces SIGINT on the whole process tree or adjusting Spack’s job tracking).

--keep-going mode

The intended behavior is that a SIGINT (or timeout) should cancel all active jobs and mark any upstream dependents as failed/skipped, preventing new launches. In the current implementation:

• Spack stops launching new jobs upon SIGINT.
• It does not reliably terminate already‑running builds, so dependent packages may stay in a running state or be left incomplete.

Thus, propagation up the dependency graph is incomplete unless all child jobs honor the forwarded SIGINT.

What to ask for

If you observe orphaned processes after Ctrl+C, please provide the output of ps -o pid,ppid,pgid,command -p <orphan‑pid> for one of the lingering compiler/make processes. That information will confirm whether the process has changed its process group or is ignoring SIGINT and will guide the recommended fix.