Lazy‑Loading vs Full‑Data Loading: Which Minimizes UI Latency Under Concurrent Cell Execution?

Problem Context

DataSpell’s Data Viewer can load tables either lazily—fetching rows on demand—or eagerly by pulling the entire DataFrame into the UI. When several notebook cells run at the same time, the choice of mode can influence how quickly the interface updates and how much memory the IDE consumes.

Constraints & Uncertainty

• Concurrent cell execution triggers multiple data‑fetch threads that may contend for the same viewer resources.
• Lazy‑loading delays the first render of a viewer, which is often hidden during single‑cell runs but can surface as a noticeable pause when several kernels start simultaneously.
• Full‑data loading removes subsequent fetch delays but can exhaust heap space, potentially causing background tasks to throttle or suspend.

Open Questions

1. Which mode yields lower overall UI latency when five cells each request a 10‑million‑row table concurrently?
2. How does memory consumption differ between the two modes under the same concurrency level?
3. Is there a table‑size threshold where lazy‑loading becomes preferable to full‑data loading in a multi‑cell scenario?