Does raising max_tis_per_query reduce Airflow burst scheduling latency, or does it move the bottleneck to the metadata database?

When a large number of task instances become runnable simultaneously, Airflow's scheduler processes them in bounded batches controlled by max_tis_per_query; at low concurrency this limit is invisible, but under a burst it can introduce queueing delay. The unresolved decision is whether increasing this batch size actually improves task start latency or simply shifts the bottleneck from the scheduler loop to the metadata database, where larger queries and longer lock hold times may degrade performance. This trade-off is further complicated by version-dependent defaults that have changed across Airflow releases and may vary by executor or deployment.

Global and per-DAG concurrency limits such as parallelism and max_active_tasks_per_dag can also hold tasks in a non-runnable state, making batch-size adjustments ineffective until those limits are raised. Executor choice changes where the queue forms—Local and Celery executors queue tasks for workers, while Kubernetes Executor creates pods per task—so identical scheduler settings can yield different latency profiles. In scheduler HA deployments, multiple schedulers may compete for the same task instances, so single-scheduler behavior may not generalize.

Which combination of max_tis_per_query, concurrency limits, and executor configuration minimizes end-to-end task start latency under sustained concurrent bursts in a production-scale metadata database? How should the batch size be validated against database CPU and lock contention without relying on remembered defaults? Does the optimal setting differ between single-scheduler and HA deployments?