ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
A practical architecture for processing large CSV files with pandas without loading the entire dataset into memory, including schema gates, incremental aggregation, and operational checks.
DuckDB's read_parquet lets you query Parquet lakes without ETL, but per-query metadata overhead and schema drift have real costs. Here's how to decide.
Stop using for-loops for element-wise array operations. Learn how NumPy broadcasting virtually expands arrays to perform high-performance vectorized math without wasting memory.
Learn how to reduce payload size and avoid rate limits by using Notion's nested filter objects and cursor-based pagination in your integration code.
Stop relying on .apply() for data transformations. Learn how to use pandas vectorization and np.where() to replace hidden loops and speed up your data pipelines by orders of magnitude.
Airflow XComs live in the metadata database by default, with size limits that fail loudly or silently depending on your setup. Here's how to decide between passing references and a custom XCom backend.
Slow ClickHouse dashboards usually mean re-aggregating raw events on every query. A materialized view feeding a well-keyed rollup table moves that work to insert time — here's a concrete setup, plus the write-overhead and backfill trade-offs.
Use a single Python file and a configuration loop to generate dozens of Airflow DAGs programmatically. This guide shows the pattern, a working TaskFlow example, scheduler verification steps, and the performance limits you'll hit at scale.
DataSpell’s Data Editor lets you edit table rows directly, auto‑generating UPDATE statements and integrating with Jupyter notebooks for a unified SQL‑Python workflow.
pandas
duckdb
python
apacheairflow
clickhouse
dataspell