Apache Airflow KubernetesExecutor: Minimal Architecture Note

Requirements

To run tasks with the KubernetesExecutor you need:

• A Kubernetes cluster version 1.21 or newer.
• Apache Airflow ≥2.0 installed with the kubernetes extra (pip install 'apache-airflow[kubernetes]').
• Network connectivity from the Airflow scheduler and webserver pods (or VMs) to the cluster’s API server.
• A service account for the scheduler that has RBAC permissions to create, get, list, and delete pods in the target namespace.

Smallest Suitable Design

A single‑node Airflow deployment can be used for evaluation:

• Webserver and scheduler run together (e.g., via the LocalExecutor for the UI and metadata database interactions).
• The scheduler is configured to use the KubernetesExecutor for all task execution.
• A minimal DAG containing a single BashOperator that runs a simple command (e.g., echo hello) is sufficient to validate the path.

Trust and Data Boundaries

Each task instance is scheduled into its own Kubernetes pod, providing process‑level isolation:

• The pod’s filesystem is ephemeral; any shared state must be persisted externally (e.g., via a PVC or external storage system).
• Secrets can be injected through Kubernetes Secrets or the Airflow Secrets Backend; connection credentials remain in the Airflow metadata database unless overridden by environment variables passed to the pod.
• The scheduler never runs the task code directly; it only communicates with the API server to manage pod lifecycle.

Operational Checks

After deploying the DAG, verify the following:

1. Scheduler logs contain the line Using KubernetesExecutor (check with kubectl logs -n airflow ).
2. When a task runs, kubectl get pods -n shows a pod whose name includes the DAG ID, task ID, and execution date (e.g., example_bash_operator_20261014T000000).
3. The pod progresses through Pending → Running → Succeeded/Failed; task logs in the Airflow UI show a source of “kubernetes” and stream the container’s stdout/stderr.
4. Resource usage (CPU/Memory) can be observed via cluster monitoring tools (e.g., kubectl top pods or a metrics server).

Failure Modes

Common failure scenarios and Airflow’s response:

• Pod eviction due to resource limits: The pod is killed; Airflow marks the task as failed and retries according to the task’s retries setting.
• Image pull errors (e.g., missing image or auth failure): The pod stays in ImagePullBackOff; the task fails and is retried.
• API server connectivity loss: The scheduler cannot create or monitor pods; tasks remain in queued state until connectivity is restored.
• RBAC misconfiguration: The scheduler receives a 403 when attempting to create pods; tasks fail immediately with an error visible in scheduler logs.
• Node failure: The pod is rescheduled to another healthy node by the Kubernetes controller; if rescheduling fails, the task is marked as failed.

Conditions That Would Change the Design

Reconsider the KubernetesExecutor when:

• Tasks require GPUs or specialized hardware – you would need node selectors, taints/tolerations, or a dedicated node pool.
• Multiple tasks need to share a persistent volume simultaneously (e.g., for large intermediate data).
• Sub‑second latency is required; pod start‑up overhead becomes dominant.
• No Kubernetes cluster is available; you would fall back to LocalExecutor or CeleryExecutor.

Concrete Example and Verification Steps

Below is a minimal DAG you can place in ~/airflow/dags/example_k8s_executor.py (adjust the path to your Airflow dags_folder).

from airflow import DAG
from airflow.operators.bash import BashOperator
from datetime import datetime, timedelta

with DAG(
    dag_id='example_k8s_executor',
    start_date=datetime(2026, 10, 14),
    schedule_interval='@once',
    catchup=False,
    default_args={'retries': 1, 'retry_delay': timedelta(minutes=2)},
) as dag:
    hello = BashOperator(
        task_id='say_hello',
        bash_command='echo "Hello from Kubernetes pod"',
    )

To verify the executor is working:

1. Trigger the DAG via the Airflow UI or CLI (airflow dags trigger example_k8s_executor).
2. On a machine with kubectl configured for the target cluster, run:

   kubectl get pods -n airflow -w
   

3. Observe a pod appear with a name like example_k8s_executor-say_hello-20261014T000000.
4. Check the pod’s logs:

   kubectl logs -n airflow <pod-name>
   

5. In the Airflow UI, open the task instance for say_hello; the log pane should show “Hello from Kubernetes pod” and indicate the log source as “kubernetes”.
6. Optionally, delete the pod manually (kubectl delete pod -n airflow <pod-name>) and confirm the task is marked as failed and then retried (if retries > 0).

Permissions needed: The service account used by the scheduler must have a role granting pods verbs create, get, list, delete in the namespace where pods are launched. Insufficient rights will cause the scheduler to log errors like Forbidden: pods is forbidden: User "system:serviceaccount:airflow:scheduler" cannot create resource "pods" in namespace "airflow".

Risks: Large custom images increase pod start‑up time, which can make short‑running tasks appear slow. Use slim base images or leverage image caching (e.g., imagePullPolicy: IfNotPresent with a local registry) to mitigate latency.

Limitations: The KubernetesExecutor does not provide low‑latency sub‑second execution; for such workloads consider a different executor or a side‑car pattern. GPU tasks require node selectors or taints/tolerations to schedule onto GPU‑enabled nodes.