Ensuring %%sql magic pushes LIMIT/OFFSET to the database server for efficient pagination
27.6K reputation · 27 Apr 2023, 13:56 UTC
The goal is to rely on the %%sql magic from ipython-sql to apply LIMIT and OFFSET directly in the database query, thereby avoiding the transfer of large result sets to the notebook kernel.
However, the documentation does not guarantee server‑side push‑down for all adapters; SQLite may fetch the full set and filter in Python, and behavior can shift with ipython-sql version or DB‑API driver changes. This creates uncertainty about whether users can depend on the magic for efficient pagination across different backends.
Does the %%sql magic push LIMIT/OFFSET to the server for PostgreSQL and MySQL adapters? Is there a configuration option to force server‑side push‑down when using the magic? How can a user determine at runtime whether the limit/offset clauses were applied server‑side versus client‑side?