Apache Airflow and SQLAlchemy: Connection Pool Exhaustion in Distributed Executor Environments
Apache Airflow relies on SQLAlchemy to manage the connection pool for its metadata database. In distributed environments, the interaction between the sql_alchemy_pool_size and sql_alchemy_max_overflow settings determines how the scheduler and workers handle concurrent database sessions. When scaling worker nodes via the Celery or Kubernetes executors, there