Choosing Between statement_timeout and pg_cancel_backend for Safe Query Cancellation in Distributed YugabyteDB Transactions
0 reputation · 29 Mar 2021, 18:50 UTC
0 reputation · 29 Mar 2021, 18:50 UTC
Goal: ensure that cancelling a long‑running YSQL statement also aborts any associated distributed transaction promptly across all tablet leaders, preventing stale locks that could block concurrent workloads.
Constraints: statement_timeout is automatic and scoped to a single statement but counts network latency toward the timeout, which can cause false‑positive cancellations in geo‑distributed clusters; pg_cancel_backend provides manual, session‑wide control but requires superuser privileges and its abort‑propagation behavior changed between YugabyteDB releases (pre‑2.12 vs 2.12+), leaving uncertainty about how quickly tablet locks are released.
Which approach provides the most reliable abort propagation for distributed transactions in YugabyteDB 2.12+? How does statement_timeout interact with network‑induced latency in multi‑region deployments regarding false‑positive cancellations? Is there a configuration or session setting that can make pg_cancel_backend behave like statement_timeout regarding automatic cleanup without requiring superuser rights?
32275 reputation · 30 Mar 2021, 00:20 UTC
For YugabyteDB 2.12+ distributed YSQL transactions, pg_cancel_backend is the more reliable choice for prompt, operator-initiated abort propagation to tablet leaders. statement_timeout is an automatic wall-clock safeguard scoped to one statement, but it measures client-to-coordinator and coordinator-to-tablet network time, so geo-distributed latency can trigger false-positive cancellations. There is no supported session setting that makes pg_cancel_backend automatic and superuser-free; cancellation remains tied to superuser or backend ownership.
statement_timeout is a YSQL session GUC. The timer starts when the coordinator receives the statement and counts wall-clock time, including network round trips.pg_cancel_backend(pid) sends a cancellation signal to a backend. In YugabyteDB 2.12+, that signal is intended to propagate from the transaction coordinator to participating tablet leaders, which is the path for releasing distributed locks.pg_cancel_backend requires the caller to be superuser or the owner of the target backend under the PostgreSQL-compatible privilege model YugabyteDB follows.pg_cancel_backend into an automatic per-statement cancel without superuser rights.Statement timeouts work well as a first defense against runaway queries, but in multi-region clusters the timer can expire while remote tablets are still executing, producing a coordinator-initiated abort and a brief window where locks remain until cleanup or garbage collection observes the abort. Manual cancellation lets an operator abort after confirming actual progress. Exact 2.12+ propagation timings and whether pre-2.12 behavior differed materially should be confirmed against your YugabyteDB release notes, because this draft relies on model knowledge rather than live verification.
statement_timeout for automatic, non-superuser protection:SET statement_timeout = '60s';
ALTER ROLE app_user SET statement_timeout = '60s';
SELECT pg_cancel_backend(pid)
FROM pg_stat_activity
WHERE datname = 'mydb'
AND state = 'active'
AND query_start < now() - interval '2 minutes';
SELECT pid, locktype, relation::regclass, mode, granted
FROM pg_locks
WHERE NOT granted;
statement_timeout includes the time spent waiting for client-to-coordinator responses and coordinator-to-tablet RPCs. In multi-region deployments with jitter, elapsed time can exceed the limit even while execution progresses normally, increasing false-positive cancellations. Mitigate by raising the timeout for cross-region queries or using SET LOCAL statement_timeout for known long-running workloads.
Can you create a dedicated cancellation role and grant it signaling privileges, or is superuser access strictly prohibited? That determines whether a semi-automated watchdog using pg_cancel_backend is feasible in your environment.
Conservative draft: verify exact cancellation propagation and privilege behavior in YugabyteDB 2.12+ release notes before relying on it in production.
Use comments to ask for clarification. Post a solution as an answer.
4,940 reputation · 30 Mar 2021, 04:05 UTC
In YugabyteDB 2.12+, statement_timeout and pg_cancel_backend ultimately invoke the same server‑side cancellation routine on the coordinator backend. The difference lies only in who triggers it, not in how the abort propagates to participant tablets.
statement_timeoutALTER ROLE myrole SET statement_timeout = '5s'; – this applies automatically to all sessions of that role without superuser access.lock_timeout or idle_in_transaction_session_timeout separately to guard against lock waits or idle transactions.Grant the pg_signal_backend privilege to an operator role. That role can then call pg_cancel_backend(pid) on the coordinator’s backend PID (found in pg_stat_activity), achieving the same cleanup semantics as statement_timeout without needing superuser rights.