ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Learn how to use Airflow's .expand() to fan out one task definition over a runtime-resolved list, with setup steps, UI verification checks, and recovery options for failed mapped instances.
DuckDB's built-in Parquet extension lets you read and write columnar files directly from SQL with automatic predicate push-down and partitioning—no Spark cluster required. Here's the practical map for engineers moving data in pipelines.
Learn how to choose between local Python interpreters and remote Jupyter kernels in DataSpell to balance compute power, data locality, and project reproducibility.
Learn how to transition Jupyter Notebooks from interactive exploration to production pipelines using nbconvert and nbclient for reproducible, headless execution.
Stop relying on Matplotlib's global state. Learn how to use the Object-Oriented API to create scalable, maintainable, and bug-free data visualizations.
A decision guide for picking an Apache Airflow executor—Sequential, Local, Celery, Kubernetes, or CeleryKubernetes— with a comparison table, real trade‑offs, and validation steps.
Airflow 2.3+ dynamic task mapping lets you create task instances at runtime from an upstream list. Example shows S3 file processing with .expand(), plus XCom size and fan‑out guardrails.
Choose between DuckDB persistent files and in-memory mode based on durability, dataset size, and concurrency constraints — with a runnable script to validate the behavior.
Learn how to use the Apache Airflow TaskFlow API to eliminate XCom boilerplate, manage data boundaries, and avoid metadata database bloat in your pipelines.
apacheairflow
duckdb
dataspell
jupyter
matplotlib