oslo.service shutdown_timeout conflict with container orchestration grace periods

Short answer

If the container runtime delivers SIGKILL before shutdown_timeout expires, the oslo.service graceful shutdown sequence simply stops existing. SIGKILL cannot be caught, blocked, or deferred by any userspace code, so the process dies immediately — mid-cleanup, mid-request, mid-database-transaction. There is no partial graceful shutdown; whatever cleanup had already run has run, and everything else is abandoned.

What actually happens, step by step

1. Kubernetes sends SIGTERM to PID 1 in the container. oslo.service's signal handler catches it and begins its graceful shutdown: it stops accepting new work, signals worker threads to finish in-flight requests, and starts counting down shutdown_timeout.
2. Kubernetes independently counts down terminationGracePeriodSeconds (default 30s). These two timers know nothing about each other.
3. Whichever timer fires first wins. If shutdown_timeout is 60s and the grace period is 30s, the kubelet instructs the runtime to SIGKILL the container at t=30s.
4. At SIGKILL, the kernel reaps the process. No Python code runs — no finally blocks, no atexit handlers, no oslo.service cleanup. In-flight RPC requests are dropped, database connections are closed by the server side only after its own keepalive/timeout detection, and any message-queue acknowledgements that were pending never happen, so those messages will be redelivered.

The practical consequence is usually not catastrophic — OpenStack's messaging layer is designed for redelivery, and most services tolerate abrupt death — but you lose exactly the guarantee shutdown_timeout was added to provide: letting long-running operations (e.g., a Nova conductor task) complete cleanly.

Recommended configuration strategy

The rule is simple: the orchestrator's grace period must be strictly larger than the application's shutdown timeout, with headroom for the SIGTERM to be delivered and for the process to actually exit after cleanup finishes.

# oslo config (e.g., nova.conf, neutron.conf)
[DEFAULT]
shutdown_timeout = 60

# Pod spec
spec:
  terminationGracePeriodSeconds: 75

A 10–15 second buffer is a reasonable starting point; measure your actual shutdown duration and adjust. If pod recycling speed matters (rollouts, node drains), the correct lever is to lower shutdown_timeout to the longest request duration you genuinely need to protect — not to inflate the grace period. A 300s shutdown timeout on a service whose longest RPC takes 20s just makes every rollout slower for no benefit.

An optional refinement is a preStop hook, but note it runs before SIGTERM and consumes the same grace-period budget, so it doesn't extend your total window — it only helps if you need to deregister the service from a load balancer or scheduler before termination begins.

Verifying the behavior in your environment

• Check the effective grace period: kubectl get pod <name> -o jsonpath='{.spec.terminationGracePeriodSeconds}'
• Delete a pod under load and watch the service logs. oslo.service logs when graceful shutdown begins and completes. If you see the "started" message but never the "completed" one, SIGKILL arrived first.
• Compare the shutdown duration in logs against both configured values and tune accordingly.

Caveats

This assumes a Victoria-or-later oslo.service with default signal handling. Custom signal handlers in a service can change the picture, and exact SIGKILL timing varies slightly across container runtimes and Kubernetes versions, so verify against your actual stack rather than trusting the defaults. Also confirm which OpenStack release you're running — pre-Victoria services ignore shutdown_timeout entirely.