Task SDK transition and XCom state management boundaries

Direct Answer

The Task SDK isolates XCom handling in a lightweight runtime that communicates with the Airflow API server rather than accessing the metadata database directly. This shifts serialization, deserialization, and large-payload offloading to the API layer, so the SDK process only holds the current task's XCom value in memory. State synchronization is bounded by the SDK's HTTP/gRPC client: pushes are committed via the API before the task reports success, and pulls are streamed on demand. Large objects bypass both the SDK and the metadata DB entirely when a custom XCom backend (S3, GCS, etc.) is configured.

Confirmed Behavior

• Memory footprint: The SDK process loads only the Airflow task SDK package and its dependencies—no SQLAlchemy ORM, no DAG parsing logic, no scheduler internals. XCom values are received as JSON-serialized payloads over the API; the SDK does not maintain a connection pool to the metadata DB.
• Push lifecycle: When a task calls xcom_push, the SDK serializes the value, sends it to the API server, and blocks until the server acknowledges persistence. The API server writes to the xcom table (or delegates to a custom backend) within the same transaction that marks the task instance state.
• Pull lifecycle: xcom_pull issues a scoped API request for the specific key(s). The response is deserialized into the task's memory and the HTTP connection is released immediately.
• Large payloads: If xcom_backend is set to an object-storage backend, the API server streams the payload directly to/from that backend. The SDK never sees the full blob; it receives a reference or a presigned URL.

Likely Explanation (Architectural Inference)

The decoupling moves the "heavy" deserialization logic (e.g., handling pickled objects, decompression, custom backend streaming) from the worker process into the API server. This prevents a single malformed or enormous XCom from OOM-killing a worker. The trade-off is added network round-trips and reliance on the API server's own memory management for concurrent large pulls.

Verification Steps

1. Run a task that pushes a 100 MB object with a custom S3 backend. Monitor the SDK worker's RSS with memory_profiler or psutil; it should stay flat.
2. Simultaneously pull the same XCom from two downstream tasks. Observe API server memory, not worker memory, for the spike.
3. Kill a task mid-push (SIGKILL). Query the metadata DB: the XCom row should be absent or marked incomplete, confirming atomicity at the API layer.

Assumptions & Uncertainty

• Assumes Airflow 2.9+ with task_sdk extra installed and AIRFLOW__CORE__ENABLE_TASK_SDK=true.
• Custom XCom backend implementation must be compatible with the API server's streaming interface; legacy backends that expect direct DB access may not work.
• Network latency between SDK and API server is unquantified here; it becomes the new bottleneck for high-frequency small XComs.

One Diagnostic Detail Needed

What is your current xcom_backend setting and average XCom payload size? This determines whether the SDK's streaming pull path or the backend's direct-download path is exercised, which changes the memory-profile recommendation.