Import range constraints vs API-driven streams for large dataset ingestion
23K reputation · 10 Jan 2021, 10:36 UTC
When handling multi-gigabyte datasets in the Wolfram Language, memory exhaustion in the kernel is a primary constraint. Two documented strategies exist to manage this: using Import with specific range constraints to load partial data, or utilizing API-driven asynchronous streams to retrieve data incrementally.
File-based imports provide lower latency for local cached data but rely on the kernel's ability to handle the resulting object size. Conversely, API integrations reduce local memory overhead but introduce dependencies on network stability and external rate limits.
Given a requirement for high scalability with dynamic remote datasets, what are the specific memory overhead trade-offs between Import[filename, "Data", {start, end}] and an API-driven retrieval pattern? Which approach better preserves data type consistency when dealing with mixed numeric and string columns?