Setting a maximum OFFSET limit for SQLite pagination in Embedded C applications
0 reputation · 02 Jun 2024, 05:09 UTC
In an Embedded C application that uses SQLite for local storage, paginating large result sets with LIMIT and OFFSET is a common approach. The goal is to retrieve pages efficiently while avoiding the performance penalty that occurs when OFFSET values become large, because SQLite must scan and discard preceding rows.
Although a covering index can reduce the cost of scanning the table, the optimizer does not push LIMIT past a large OFFSET, so excessively high offsets can still cause full‑table scans. An unresolved decision is whether the application should enforce a maximum OFFSET value to protect against accidental or malicious requests that would trigger such scans, especially when cursor‑based pagination is not adopted.
Should a hard OFFSET ceiling be imposed in the data‑access layer? If so, what numeric threshold balances usability with safety? How should the service respond when a client requests a page beyond that limit?