Airflow XComs and the Metadata Database: Why Your Tasks Should Pass Keys, Not Data

Your DAG ran green for weeks. Then an upstream API started returning bigger payloads, and now a downstream task is failing with a database error that mentions neither XCom nor size limits — or worse, the task silently succeeds while the value it pushed never actually lands. The culprit is almost always the same: someone treated XCom as a data channel instead of a message channel.

The short version: by default, every XCom value is serialized and written as a row in Airflow's metadata database. That database is designed for scheduler state, not bulk payloads, and it imposes hard size limits that vary by backend. The fix is not to raise the limit — it's to stop pushing data through XCom at all, or to move XCom storage to object storage with a custom backend.

What actually happens when you push an XCom

Calling ti.xcom_push(key="result", value=payload) — or returning a value from a @task function, which pushes implicitly — serializes the value (JSON by default) and inserts it into the xcom table in the metadata database. Every pull reads it back from the same table.

The size ceiling depends on your database. On MySQL, the value column is a BLOB capped at 64KB; on PostgreSQL it's effectively unbounded at the column level but still a terrible idea at scale. The commonly cited ~48KB figure comes from MySQL deployments where oversized values are rejected or truncated. The failure mode is the dangerous part: depending on the database and strictness settings, you can get a loud insert error, a truncated value that deserializes into garbage, or a downstream task that pulls None and keeps going.

Even when values fit, big XComs hurt. The scheduler and webserver query the metadata DB constantly, and every worker pulling a 40KB blob adds load to a database whose real job is tracking task state. I've seen the advice "just use Postgres, it handles it" — technically true, practically a slow-motion outage.

The decision: pass references, not payloads

The robust pattern is simple: the producing task writes the data somewhere durable (S3, GCS, a scratch table) and pushes only the pointer — a path, a URI, a key — through XCom. The consuming task pulls the pointer and fetches the data itself.

from airflow.decorators import dag, task
import pendulum

@dag(start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), schedule="@daily", catchup=False)
def reference_pattern():

    @task
    def extract() -> dict:
        # Write the heavy payload to object storage yourself
        path = "s3://my-bucket/extracts/{{ ds }}/records.parquet"
        write_records_to_s3(path)  # your helper
        return {"path": path, "row_count": count_records()}

    @task
    def transform(ref: dict):
        df = read_parquet(ref["path"])
        # ... work on the data ...

    transform(extract())

reference_pattern()

The XCom here is a few hundred bytes of JSON — metadata about the data, not the data. This survives payload growth, keeps the metadata DB lean, and gives you a natural place to enforce retention (lifecycle rules on the bucket).

If you must push data: a custom XCom backend

Sometimes you can't restructure the tasks — legacy DAGs, third-party operators that push results automatically. Airflow 2.x supports pluggable XCom backends via the AIRFLOW__CORE__XCOM_BACKEND environment variable (or xcom_backend in airflow.cfg). The standard approach is a threshold-based backend: small values stay in the metadata DB, large ones are offloaded to object storage and replaced with a reference string.

# airflow_local_settings.py or a module on PYTHONPATH
from airflow.models.xcom import BaseXCom
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
import json
import uuid

THRESHOLD = 40 * 1024  # bytes, below the DB limit with margin
BUCKET = "my-xcom-bucket"
PREFIX = "xcom"

class S3XComBackend(BaseXCom):
    @staticmethod
    def serialize_value(value, **kwargs):
        payload = json.dumps(value).encode()
        if len(payload) > THRESHOLD:
            key = f"{PREFIX}/{uuid.uuid4().hex}.json"
            S3Hook().load_bytes(payload, key=key, bucket_name=BUCKET, replace=True)
            value = {"__xcom_s3__": f"s3://{BUCKET}/{key}"}
        return BaseXCom.serialize_value(value, **kwargs)

    @staticmethod
    def deserialize_value(result):
        value = BaseXCom.deserialize_value(result)
        if isinstance(value, dict) and "__xcom_s3__" in value:
            uri = value["__xcom_s3__"]
            bucket, key = uri.replace("s3://", "").split("/", 1)
            raw = S3Hook().read_key(key=key, bucket_name=bucket)
            return json.loads(raw)
        return value

Set AIRFLOW__CORE__XCOM_BACKEND=airflow_local_settings.S3XComBackend (adjust the module path to wherever the class lives) on every component — scheduler, workers, webserver, triggerer. If any component runs without the backend configured, it will deserialize the reference dict as a literal value and downstream tasks get {"__xcom_s3__": ...} instead of their data. This is the most common deployment mistake.

Before rolling this out, verify the round trip: push a value above your threshold from a test task, confirm an object appears in the bucket, and confirm a downstream pull returns the original structure. Also check that your workers' IAM role or connection has read/write on the bucket — permission failures here surface as task errors, not config errors.

Trade-offs worth knowing upfront

A custom backend is not free. Every large XCom now costs a network round trip on push and pull, which adds latency and a new failure surface (bucket unavailable, credentials expired). Orphaned objects accumulate if tasks fail mid-DAG or old XComs are purged from the database — the DB cleanup doesn't know about your S3 keys, so you need a lifecycle policy on the bucket. And mixing backends across an upgrade (values written by the old backend, read by the new one) can break in-flight DAG runs, so time the switch for a quiet window.

The reference-passing pattern avoids all of this but requires discipline: every task author has to know the convention, and implicit pushes from operators can still leak big values into the DB.

Where to start

Audit before you change anything. Query the xcom table for the largest values — SELECT dag_id, task_id, length(value) FROM xcom ORDER BY length(value) DESC LIMIT 20; works on both MySQL and Postgres (run it read-only against the metadata DB; on a busy production instance, prefer a replica). If the top values are under a few KB, you have a hygiene problem, not an infrastructure one — fix the tasks. If you routinely see tens of KB, decide: restructure to pass references where you control the code, and deploy a threshold backend as a safety net for what you don't.