Does a client-side timeout cancel in-flight work in OpenStack's Nova, Neutron, and Cinder?

Question

OpenStack has no request-wide cancellation token or deadline propagation standard shared across its services. When a client times out or disconnects, work already accepted by Nova, Neutron, or Cinder generally continues server-side unless it is explicitly aborted through a service-specific action such as Nova's delete or force-delete.

The open question is how deadlines compose across layers. Timeouts are configured per service through oslo.config options such as rpc_response_timeout, and the effective limits can differ between the REST API layer, the message queue, and the driver or hypervisor. Whether an explicit abort actually halts a build or attach depends on driver support and the current task state, and both defaults and abort semantics vary between releases, so end-to-end behavior is not guaranteed to be consistent.

Does a client-side timeout or disconnect cancel in-flight server-side work, or does the operation continue and require later reconciliation?
Which configuration options determine the effective end-to-end deadline for a multi-step operation across the API, RPC, and driver layers?
Where are partially created resources expected to be cleaned up when no abort path succeeds?

Tasadduq Burney · Answer

Direct AnswerA client-side timeout or disconnect does not automatically cancel in-flight work in Nova, Neutron, or Cinder. The RPC call continues executing server-side unless an explicit abort action is invoked through a service-specific API (e.g., Nova delete/force-delete, Neutron port delete, Cinder volume detach/force-delete). Even then, the abort only takes effect if the underlying driver supports cancellation and the operation is in a cancellable state.Why the Work ContinuesOpenStack services communicate via a message queue (typically RabbitMQ or Qpid) using oslo.messaging. When a REST API request arrives, the API service publishes an RPC call to the queue. The client's HTTP timeout governs only how long the API service waits for an RPC reply (rpc_response_timeout). If that timeout expires, the API returns an error to the client, but the consumer (e.g., nova-compute, neutron-server, cinder-volume) has already dequeued the message and begun processing. There is no standard deadline-propagation or cancellation token that reaches the worker.Configuration Options That Shape the Effective DeadlineThe end-to-end deadline is a composition of per-service timeouts, not a single value:API layer: rpc_response_timeout (default 60s in many releases) in nova.conf, neutron.conf, cinder.conf — how long the API service waits for an RPC reply.Message queue: RabbitMQ heartbeat and publisher confirms; no built-in TTL for RPC requests.Worker/driver layer: Driver-specific timeouts (e.g., libvirt connection timeout, storage backend timeout) and task-state machines that may or may not honor an abort request.Because these layers are configured independently, a 30-second client timeout can coexist with a 300-second driver operation.Explicit Abort Paths and Their LimitsServiceAbort APIConditions for EffectivenessNovaDELETE /servers/{id} or force-deleteInstance must be in a state where the compute driver can interrupt the task (e.g., BUILD, REBUILD). force-delete bypasses the conductor but still requires driver cooperation.NeutronDELETE /ports/{id}, DELETE /networks/{id}Agent-side cleanup (L2/L3/DHCP agents) must be reachable; if the agent is down, the resource lingers until the agent recovers or manual cleanup runs.CinderDELETE /volumes/{id} with force=trueVolume must not be attached; backend driver must implement delete_volume and honor the request. Some backends (e.g., Ceph RBD) may leave orphaned images if the driver call fails.If the driver does not implement cancellation or the task has passed a point of no return (e.g., disk image copy in progress, hypervisor domain creation committed), the abort request is logged but the work completes.Cleanup of Partially Created ResourcesWhen no abort succeeds, orphaned resources are handled by periodic tasks:Nova: nova-compute runs _cleanup_incomplete_migrations and instance power-state reconciliation; nova-conductor can reap ERROR state instances.Neutron: L2/L3/DHCP agents reconcile ports and networks on restart; neutron-server has DB cleanup cron jobs for stale bindings.Cinder: cinder-volume manager runs _cleanup_volumes and _cleanup_snapshots; backend-specific orphan cleanup (e.g., cinder-manage db purge) may be required.These periodic tasks run on configurable intervals (often 300–600 seconds), so orphaned resources can persist for minutes to hours.Verification StepsCheck current timeout settings:grep -E 'rpc_response_timeout|rpc_cast_timeout' /etc/nova/nova.conf /etc/neutron/neutron.conf /etc/cinder/cinder.confTrigger a long-running operation (e.g., boot a large instance, create a large volume) and impose a short client timeout (curl --max-time 5 or a low SDK timeout).Monitor service logs (/var/log/nova/nova-compute.log, neutron-server.log, cinder-volume.log) to confirm the worker continues after the client receives a timeout error.Attempt an explicit abort (openstack server delete --force, openstack volume delete --force) and observe whether the task stops or completes.Assumptions and UncertaintyBehavior described applies to OpenStack releases from Train (2019) through 2024.1 (Caracal). Older releases (pre-Stein) may lack force-delete or have different default timeouts.Driver support for cancellation varies: libvirt generally honors domainDestroy during BUILD; VMware, Hyper-V, and proprietary storage drivers may not. This is a review required item for your specific driver stack.Message queue configuration (publisher confirms, message TTL) can introduce additional latency but does not provide cancellation semantics.One Diagnostic Detail That Changes the RecommendationWhich hypervisor/storage drivers are deployed for Nova compute, Neutron agents, and Cinder backends? Driver-specific abort support is the single factor that determines whether an explicit delete actually halts in-flight work.

Does a client-side timeout cancel in-flight work in OpenStack's Nova, Neutron, and Cinder?

1 answer

Direct Answer

Why the Work Continues

Configuration Options That Shape the Effective Deadline

Explicit Abort Paths and Their Limits

Cleanup of Partially Created Resources

Verification Steps

Assumptions and Uncertainty

One Diagnostic Detail That Changes the Recommendation

Your perspective belongs here.

0 question comments

Service	Abort API	Conditions for Effectiveness
Nova	`DELETE /servers/{id}` or `force-delete`	Instance must be in a state where the compute driver can interrupt the task (e.g., `BUILD`, `REBUILD`). `force-delete` bypasses the conductor but still requires driver cooperation.
Neutron	`DELETE /ports/{id}`, `DELETE /networks/{id}`	Agent-side cleanup (L2/L3/DHCP agents) must be reachable; if the agent is down, the resource lingers until the agent recovers or manual cleanup runs.
Cinder	`DELETE /volumes/{id}` with `force=true`	Volume must not be `attached`; backend driver must implement `delete_volume` and honor the request. Some backends (e.g., Ceph RBD) may leave orphaned images if the driver call fails.