Efficient Notion API Queries: Combining Nested Filters with Cursor Pagination
Learn how to reduce payload size and avoid rate limits by using Notion's nested filter objects and cursor-based pagination in your integration code.
Loading…
ReadMeFeed / Community knowledge
#data-engineering
Real questions. Useful conversations. Find the people who know your stack.
Stop Using .apply(): Mastering Pandas Vectorization for Speed
Stop relying on .apply() for data transformations. Learn how to use pandas vectorization and np.where() to replace hidden loops and speed up your data pipelines by orders of magnitude.
pandas
pythonAirflow XComs and the Metadata Database: Why Your Tasks Should Pass Keys, Not Data
Airflow XComs live in the metadata database by default, with size limits that fail loudly or silently depending on your setup. Here's how to decide between passing references and a custom XCom backend.
apacheairflowWhen ClickHouse Feels Slow, the Answer Is Often a Materialized View
Slow ClickHouse dashboards usually mean re-aggregating raw events on every query. A materialized view feeding a well-keyed rollup table moves that work to insert time — here's a concrete setup, plus the write-overhead and backfill trade-offs.
clickhouseScaling Airflow Pipelines with Loop-Based Dynamic DAG Generation
Use a single Python file and a configuration loop to generate dozens of Airflow DAGs programmatically. This guide shows the pattern, a working TaskFlow example, scheduler verification steps, and the performance limits you'll hit at scale.
DataSpell’s Integrated Data Editor: Cut SQL Boilerplate and Speed Up Validation
DataSpell’s Data Editor lets you edit table rows directly, auto‑generating UPDATE statements and integrating with Jupyter notebooks for a unified SQL‑Python workflow.
dataspellImplementing Dynamic Task Mapping in Apache Airflow
Learn how to use Airflow's .expand() to fan out one task definition over a runtime-resolved list, with setup steps, UI verification checks, and recovery options for failed mapped instances.
DuckDB's Native Parquet Support: What Engineers Actually Need to Know
DuckDB's built-in Parquet extension lets you read and write columnar files directly from SQL with automatic predicate push-down and partitioning—no Spark cluster required. Here's the practical map for engineers moving data in pipelines.
duckdbChoosing Between Local Interpreters and Remote Jupyter Kernels in DataSpell
Learn how to choose between local Python interpreters and remote Jupyter kernels in DataSpell to balance compute power, data locality, and project reproducibility.