ReadMeFeed / Community knowledge
Real questions. Useful conversations. Find the people who know your stack.
Choose partitioned Parquet writes from DuckDB when downstream reads filter on a low‑cardinality column. Otherwise a single file reduces overhead. This guide compares trade‑offs and shows validation steps.
Learn how to create time-unit and integer-range partitioned tables in BigQuery, enable require_partition_filter, and verify partition pruning to lower query costs and avoid full table scans.
Eliminate context switching by using DataSpell's unified database engine to manage schemas, perform visual data edits, and generate ER diagrams within your IDE.
Spark broadcast joins avoid shuffling the large side of a join when one dataset fits in memory. This post walks through the configuration, a concrete example, and how to verify the improvement.
Stop relying on print() statements. Learn how DataSpell's Interactive Console and Variable View decouple execution state from notebook cells to prevent the 'state trap'.
ShotGrid's find_all() method works for small datasets but fails at scale. Learn to implement paginated fetching with limit and page parameters to keep memory flat and avoid timeouts when processing tens of thousands of production entities.
Detaspace’s native Delta Lake support lets you add columns on‑the‑fly with minimal overhead. Learn how to evolve schemas, the limits of the feature, and a step‑by‑step example that keeps your lake consistent and performant.
Compare Snappy, Zstd, and Gzip for Parquet files in Detaspace. Learn constraints, trade-offs, and a validation workflow for selecting the right compression algorithm.
Stop R from crashing on large datasets by using data.table's := operator for in-place modification, avoiding the memory overhead of copy-on-modify.
google
dataspell
shotgrid
detaspace
r