Cairo query cancellation: does a hard timeout during shuffle guarantee intermediate state cleanup?
0 reputation · 15 Nov 2021, 11:14 UTC
Cairo query cancellation and intermediate state cleanup
In Cairo’s distributed execution model, a cancellation request is propagated from the coordinator to all worker nodes via a de‑registration signal. While active scans typically stop immediately, long‑running transformation stages may continue until a checkpoint is reached. When a hard timeout is enforced by a client‑side or gateway configuration, it is unclear whether the execution engine guarantees the immediate release of intermediate shuffle data.
Given that timeouts are often set in application drivers or server‑side parameters, the consistency of resource cleanup during a mid‑shuffle timeout remains an open question. This uncertainty affects memory pressure on worker nodes and the predictability of query termination.
What is the documented behavior of Cairo when a hard timeout occurs mid‑shuffle? Is there an execution‑engine parameter that forces immediate cleanup of intermediate state, or does cleanup only happen at the next checkpoint? How does the current configuration transition from session‑based to query‑level timeouts influence this behavior?