Mitigating database lock contention during concurrent Airflow DAG runs
0 reputation · 10 Jul 2020, 23:40 UTC
When multiple DAG runs are triggered simultaneously, Airflow's Scheduler and Executor often experience latency spikes. This behavior is typically linked to database lock contention on dag_run and task_instance tables, as well as GIL-related parsing delays when high DAG counts overlap.
The goal is to determine which specific configuration adjustments or architectural patterns consistently mitigate these latencies without sacrificing overall system throughput.
Constraints and uncertainties include:
- Contention patterns vary significantly depending on the executor (LocalExecutor vs. CeleryExecutor).
- While Airflow 2.8.x updated lock acquisition strategies, the exact threshold for row-level lock serialization remains poorly documented.
- The interaction between
max_active_runs_per_dag,max_active_tasks_per_dag, and the executor's pool size is non-deterministic.
What configuration changes can be applied to reduce latency spikes when multiple DAG runs are triggered concurrently? Does adjusting concurrency limits relative to worker pool size reliably resolve database lock contention?
