Choosing an XCom Backend for Apache Airflow: Architecture Note

Requirements

Airflow DAGs often need to pass small to medium-sized data between tasks via XCom. The chosen backend must:

• Provide low-latency read/write for frequent task communication.
• Scale with the number of task instances without becoming a bottleneck.
• Preserve data integrity across executor restarts and worker failures.
• Respect trust boundaries: data should not be exposed to unauthorized components.

Smallest Suitable Design

For low‑volume deployments (few hundred task instances per day) the default relational‑database backend is sufficient. It stores XCom values in the xcom table of the Airflow metadata database (PostgreSQL/MySQL). No extra services are required, and the backend is automatically available after a standard Airflow install.

Trust/Data Boundaries

When a task pushes an XCom value, the worker serializes the payload and writes it to the backend. The scheduler or another worker later reads the value. Therefore the backend must be accessible from:

• All worker nodes (Celery, KubernetesExecutor, LocalExecutor).
• The scheduler process (for trigger‑based XCom pulls).
• Any CLI or UI process that runs airflow xcom show.

If the backend is a shared service (e.g., Redis), network policies should restrict access to the Airflow namespace only. For a filesystem backend, the storage must be mounted identically on every worker and the scheduler.

Operational Checks

Verify the Active Backend

Inspect the configuration and compare it with the metadata stored in the xcom table:

# On the Airflow host (requires read access to airflow.cfg and DB)
grep ^xcom_backend $AIRFLOW_HOME/airflow.cfg
# Example output: xcom_backend = airflow.models.xcom.DefaultXComBackend

# Check what the DB thinks is stored (requires DB read access)
psql -d airflow -c "SELECT DISTINCT backend FROM xcom LIMIT 5;"

The two values should match. A mismatch indicates that the backend was changed without restarting components, which can lead to stale connection pools.

Measure Latency

Use the CLI to fetch a known XCom key and note the response time:

# Replace with a real task_instance_key (dag_id, task_id, execution_date, key)
airflow xcom show example_bash_task run_id_0 push_value

With the default DB backend you will see a query latency proportional to DB load. With a Redis backend the return is typically sub‑millisecond, assuming Redis is healthy and reachable.

Failure Modes

• Database bottleneck: High-frequency XCom pushes (>10k per hour) increase write load, cause autovacuum pressure, and can slow scheduler heartbeat queries.
• Filesystem locality loss: In executor‑based clusters (KubernetesExecutor, CeleryExecutor) a local‑disk XCom backend creates files on the worker’s ephemeral storage. If the scheduler runs on a different node, airflow xcom show returns None because the file is missing.
• Silent corruption: Misconfigured TLS or authentication on Redis can cause workers to fallback to empty values without raising an error, leading downstream tasks to receive None or corrupted data.
• Backend incompatibility: Switching from DB to Redis (or vice‑versa) without migrating existing XCom rows leaves rows in the old format; reads return deserialization errors or empty values.

Conditions That Would Change the Design

• Throughput exceeds ~10k task instances per day or average XCom payload > 1 MB → consider Redis or a dedicated object store (e.g., S3 via a custom backend).
• Executor type is KubernetesExecutor and workers are short‑lived pods → a shared filesystem backend requires a ReadWriteMany PersistentVolumeClaim; otherwise use Redis.
• Security policy forbids exposing the metadata DB to worker nodes → move XCom to a side‑car Redis instance with network‑level isolation.
• Operational team lacks Redis expertise → stay with the DB backend and mitigate bottlenecks by archiving old XCom rows periodically.

Practical Example: Switching to Redis Backend

1. Prepare Redis: Deploy a Redis instance with TLS enabled and set a strong password. Ensure it is accessible from all worker and scheduler pods.
2. Update airflow.cfg:

   [core]
   xcom_backend = airflow.models.xcom.XComBackend
   # Point to the Redis backend class
   xcom_backend_kwargs = {"host": "redis-airflow.internal", "port": 6379, "password": "${REDIS_PASSWORD}", "ssl": true}
   

3. Restart components: Roll a restart of the scheduler, all workers, and the webserver to clear any cached connections.
4. Validate: Run the verification steps above. Confirm the backend column in the xcom table shows RedisXComBackend (or the custom class name) and that airflow xcom show returns instantly.
5. Monitor: Track Redis metrics (memory usage, hit/miss ratio) and Airflow scheduler latency for at least one full DAG run cycle.

Limitations and How to Check Them

Even with a well‑chosen backend, operators should:

• Check for orphaned XCom rows after a backend switch: SELECT COUNT(*) FROM xcom WHERE backend IS NULL; – any non‑zero count indicates missing backend metadata.
• Set a retention policy (e.g., delete XCom older than 30 days) to prevent unbounded growth in the DB or Redis.
• Test failure scenarios in a staging cluster: simulate a Redis network partition and verify that tasks fail loudly (raise AirflowException) rather than silently proceeding with None values.