ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Apache Airflow's TaskFlow API replaces operator boilerplate with plain Python functions, automating XCom handling and enabling dynamic task mapping via .expand(). This blog shows a concrete DAG example, explains executor-agnostic behavior, and highlights metadata-load limits.
Airflow XComs live in the metadata database by default, with size limits that fail loudly or silently depending on your setup. Here's how to decide between passing references and a custom XCom backend.
A concise architecture note for running Apache Airflow tasks with the KubernetesExecutor, covering requirements, minimal design, trust boundaries, operational checks, failure modes, and design change triggers.
Use a single Python file and a configuration loop to generate dozens of Airflow DAGs programmatically. This guide shows the pattern, a working TaskFlow example, scheduler verification steps, and the performance limits you'll hit at scale.
Learn how to use Airflow's .expand() to fan out one task definition over a runtime-resolved list, with setup steps, UI verification checks, and recovery options for failed mapped instances.
A nightly pipeline must process an unknown number of files. Airflow's dynamic task mapping (.expand(), stable since 2.3) creates one real task instance per item at runtime — with per-item retries, logs, and UI status.
A decision guide for picking an Apache Airflow executor—Sequential, Local, Celery, Kubernetes, or CeleryKubernetes— with a comparison table, real trade‑offs, and validation steps.
Pass data between Airflow tasks with XComs—understand limits, register custom serializers, and choose between in‑memory and external storage for optimal performance.
Airflow 2.3+ dynamic task mapping lets you create task instances at runtime from an upstream list. Example shows S3 file processing with .expand(), plus XCom size and fan‑out guardrails.
python
apacheairflow