ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Guidance on selecting and operating an XCom backend in Apache Airflow, covering requirements, minimal design, trust boundaries, checks, failure modes, and when to change the design.
Airflow XComs live in the metadata database by default, with size limits that fail loudly or silently depending on your setup. Here's how to decide between passing references and a custom XCom backend.
Pass data between Airflow tasks with XComs—understand limits, register custom serializers, and choose between in‑memory and external storage for optimal performance.
Learn how the @task decorator turns Python functions into tasks with automatic XCom handling, making DAGs cleaner and easier to maintain.
Learn how to use the Apache Airflow TaskFlow API to eliminate XCom boilerplate, manage data boundaries, and avoid metadata database bloat in your pipelines.
Learn how switching Airflow’s XCom backend from the default database to Redis or a shared filesystem reduces metadata‑DB load, avoids pagination bottlenecks, and what trade‑offs to consider.
Learn how the @task decorator turns Python functions into Airflow tasks with automatic XCom handling, see a concrete ETL example, and understand size limits and common pitfalls.
Learn how the TaskFlow API lets you write tasks as plain Python functions, automatically handling dependencies and data passing, and understand its pickle‑based limitations.
Apache Airflow is transitioning toward a decoupled execution model via the Task SDK to reduce memory overhead by separating task execution logic from the core environment. This architectural shift aims to mitigate resource leaks associated with loading the full Airflow core in every worker process. A specific uncertainty exists regarding the state management
apacheairflow
python