Practical Guide to Using Airflow XComs for Inter‑Task Data Transfer
Pass data between Airflow tasks with XComs—understand limits, register custom serializers, and choose between in‑memory and external storage for optimal performance.
Loading…
ReadMeFeed / Community knowledge
#apache-airflow
Real questions. Useful conversations. Find the people who know your stack.
Dynamic Task Mapping in Apache Airflow: Replace Static Loops with Runtime Parallelism
Airflow 2.3+ dynamic task mapping lets you create task instances at runtime from an upstream list. Example shows S3 file processing with .expand(), plus XCom size and fan‑out guardrails.
apacheairflowUsing Airflow’s TaskFlow API to Replace Manual XCom Boilerplate
Learn how the @task decorator turns Python functions into tasks with automatic XCom handling, making DAGs cleaner and easier to maintain.
Architecting Data Flow in Airflow: Transitioning to the TaskFlow API
Learn how to use the Apache Airflow TaskFlow API to eliminate XCom boilerplate, manage data boundaries, and avoid metadata database bloat in your pipelines.
Choosing the Right Airflow Executor for Your Workload
A decision guide for selecting an Apache Airflow executor (Local, Celery, Kubernetes, CeleryKubernetes) based on workload scale, infrastructure constraints, and operational trade-offs, with configuration examples and validation steps.
Using Apache Airflow TaskFlow API to Simplify DAG Definition
Learn how the @task decorator turns Python functions into Airflow tasks with automatic XCom handling, see a concrete ETL example, and understand size limits and common pitfalls.
Using Apache Airflow TaskFlow API to Eliminate XCom Boilerplate
Learn how the TaskFlow API lets you write tasks as plain Python functions, automatically handling dependencies and data passing, and understand its pickle‑based limitations.
pythonDiagnosing Apache Airflow Scheduler Deadlocks that Keep Tasks in Queued State
Identify why Airflow tasks stay queued due to scheduler deadlocks, check logs, DB transactions, and process counts, then apply targeted fixes and know when to escalate.
Scaling Pipeline Deployment with Dynamic DAG Generation in Apache Airflow
Stop copy-pasting DAG files. Learn how to use dynamic DAG generation in Apache Airflow to scale your pipelines using configuration files while avoiding Scheduler performance pitfalls.