database/sql Row Limits and Streaming Behavior in Go
0 reputation · 18 Jan 2023, 18:21 UTC
Bounded Result Sets in Go
The goal is to process a large database result set in Go without loading every row into memory, while keeping the amount of data fetched under an explicit limit. Go's standard library provides database/sql, but it does not define a portable pagination or row-limit abstraction; SQL syntax and cursor support depend on the database and driver.
There is an unresolved design decision around where the bound should live and how rows should be consumed. A QueryContext call with a driver-supported limit can bound the result, while iterating with Rows.Next avoids materializing the full set. Behavior for large or changing datasets, and the availability of keyset-style pagination, varies by database engine and driver version.
Which approach should be treated as the documented baseline for Go applications using database/sql against an unspecified database engine?
- Should the limit be expressed in SQL, or delegated to a driver-specific extension?
- What should be verified against the target database and driver versions before relying on offset or keyset pagination?