Structured concurrency scope bounds during zero-downtime service migration
0 reputation · 17 May 2022, 22:11 UTC
0 reputation · 17 May 2022, 22:11 UTC
A small Kotlin service is being migrated to a new version using a rolling deployment where old and new instances coexist briefly. The migration tasks run as suspend functions on Dispatchers.IO within a CoroutineScope tied to the service lifecycle component.
Ensure that all in-flight migration coroutines — including blocking I/O offloaded to Dispatchers.IO — terminate cleanly when the old instance receives a shutdown signal, without leaking resources or leaving orphaned operations on the shared thread pool.
Structured concurrency guarantees that child coroutines complete or are cancelled before the parent scope finishes. However, the documentation does not explicitly clarify whether this guarantee extends across process boundaries during a rolling update, nor how CancellationException propagates into blocking calls already executing on Dispatchers.IO. Using GlobalScope is ruled out due to leak risk, but the lifecycle scope may be destroyed before the dispatcher finishes interruptible I/O.
CoroutineScope cancellation reliably interrupt blocking operations running on Dispatchers.IO when the process is terminating?Job.join() completes on the dispatcher threads?35275 reputation · 18 May 2022, 02:40 UTC
1. Does a parent CoroutineScope cancellation reliably interrupt blocking operations running on Dispatchers.IO when the process is terminating?
No. Cancellation marks the coroutine’s Job as cancelled, but the underlying thread will only stop the blocking call if that call is interruptible (e.g., uses Thread.interrupt() or a library that checks the interrupted flag). Pure blocking I/O such as java.io.InputStream.read() or Thread.sleep() will continue until it naturally returns, even though the coroutine is cancelled. Therefore, a parent scope cancellation does not guarantee that a blocking operation on Dispatchers.IO will be aborted immediately during a process shutdown.
2. What is the expected behavior of child coroutines if the JVM shuts down before Job.join() completes on the dispatcher threads?
During a JVM shutdown, the runtime does not wait for unfinished coroutines. All non‑daemon threads, including those in Dispatchers.IO, are terminated abruptly. Any finally blocks or cleanup code in the coroutine may be skipped, and resources opened by the coroutine could remain leaked until the process exits. Consequently, if Job.join() is not completed before the JVM exits, the child coroutine’s cleanup logic may never run.
Job to all child jobs.java.nio.channels.SocketChannel.read, URLConnection.connect with timeouts) honour Thread.interrupt() and will throw InterruptedIOException when the coroutine is cancelled.Job.join(); it terminates all threads after the shutdown hooks finish.Wrap blocking work in withContext(Dispatchers.IO) and use libraries that support interruption. If you must use a blocking API that does not support interruption, consider running it in a dedicated thread pool that you can shut down explicitly.
Register a shutdown hook that cancels the service’s lifecycle scope.
Runtime.getRuntime().addShutdownHook(Thread {
serviceScope.cancel("shutdown signal received")
// Give coroutines a bounded time to finish
runBlocking {
withTimeoutOrNull(5000L) { serviceScope.coroutineContext[Job]?.join() }
}
})
Use SupervisorJob if you want child failures not to cancel the entire scope.
Instrument cleanup blocks. Log entry and exit of finally blocks so you can verify whether they run before the JVM exits.
Test the shutdown path locally. Create a coroutine that performs a blocking read, cancel the scope, and observe whether the thread’s interrupted flag is set and whether the operation stops. Repeat with both interruptible and non‑interruptible APIs.
Could you specify which blocking I/O APIs the migration tasks use (e.g., java.io.InputStream.read, java.nio.channels.SocketChannel.read, or a custom blocking library)? Knowing whether those APIs honour thread interruption will determine whether the above shutdown hook alone is sufficient or if you need a dedicated thread pool that you shut down explicitly.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.