Question
Docker Compose CeleryExecutor to KubernetesExecutor local task-execution parity
Cig Maple
0 reputation · 15 Aug 2024, 08:11 UTC
117.7K views1
Loading…
0 reputation · 15 Aug 2024, 08:11 UTC
Goal: determine whether a repeatable local Docker Compose stack can faithfully reproduce production task-execution semantics when the executor is changed from CeleryExecutor to KubernetesExecutor.
The official Airflow Docker Compose reference is architected around CeleryExecutor, with scheduler, worker, triggerer, Redis, and Postgres services. KubernetesExecutor cannot be fully exercised in that same stack because it requires a Kubernetes API server and pod lifecycle control, typically an external kind or minikube cluster. Airflow's DAG parsing and task-instance state machine are executor-agnostic, but worker autoscaling, retry isolation, resource limits, and log handling can diverge, producing environment drift between local Compose and production Kubernetes.
Assume an Airflow 2.x reference image and the documented Compose service layout for that tag. Executor defaults and service definitions change across releases, so the exact image version must be fixed before comparing behavior. Mounted-volume permission handling for DAGs, logs, and plugins may also appear differently on Celery workers versus Kubernetes pods.
35775 reputation · 15 Aug 2024, 18:39 UTC
DAG parsing and task-instance state machine logic remain consistent regardless of whether CeleryExecutor or KubernetesExecutor is used in Airflow.
A local Docker Compose stack cannot faithfully reproduce KubernetesExecutor semantics because it lacks the required Kubernetes API server and pod lifecycle management. Differences in retry isolation, log aggregation, and resource enforcement are environmental, not purely Airflow configuration issues.
Normalizing mounted-volume permission handling between local Compose and Kubernetes pods is highly dependent on the underlying OS and container runtime configuration, not solely Airflow settings.
Use comments to ask for clarification. Post a solution as an answer.
35,775 reputation · 15 Aug 2024, 14:18 UTC
Assuming Airflow 2.x, swapping executors also changes scheduler concurrency defaults: task_parallelism is CeleryExecutor-specific and has no direct KubernetesExecutor equivalent, while parallelism and max_active_tasks_per_core apply globally but interact differently with pod scheduling. A local Compose stack can mask this because Celery workers share a fixed process pool, whereas KubernetesExecutor provisions pods on demand. Verify parity by running airflow config get-value core parallelism and airflow config get-value celery task_parallelism under each executor, then backfill the same DAG and compare concurrent task-instance counts.
apacheairflow