database/sql Row Limits and Streaming Behavior in Go
0 reputation · 18 Jan 2023, 18:21 UTC
0 reputation · 18 Jan 2023, 18:21 UTC
The goal is to process a large database result set in Go without loading every row into memory, while keeping the amount of data fetched under an explicit limit. Go's standard library provides database/sql, but it does not define a portable pagination or row-limit abstraction; SQL syntax and cursor support depend on the database and driver.
There is an unresolved design decision around where the bound should live and how rows should be consumed. A QueryContext call with a driver-supported limit can bound the result, while iterating with Rows.Next avoids materializing the full set. Behavior for large or changing datasets, and the availability of keyset-style pagination, varies by database engine and driver version.
Which approach should be treated as the documented baseline for Go applications using database/sql against an unspecified database engine?
35775 reputation · 18 Jan 2023, 23:09 UTC
Treat the row limit as part of the SQL statement, not as a database/sql API feature. Pass a query containing the engine's row-limit clause—typically LIMIT or FETCH FIRST—to QueryContext, then consume the result set incrementally with Rows.Next. Close Rows as soon as iteration ends and check Rows.Err after the loop. Do not load all rows into a slice, and do not default to driver-specific extensions unless the dialect lacks the needed clause.
Because SQL syntax differs by engine, the standard library cannot enforce a single row-limit abstraction. The bound lives in the query text; database/sql only transports it. Streaming behavior depends on the driver, so a query that streams on one engine may buffer on another.
rows, err := db.QueryContext(ctx, `
SELECT id, name FROM events
ORDER BY created_at, id
LIMIT ? OFFSET ?
`, limit, offset)
if err != nil {
return err
}
defer rows.Close()
for rows.Next() {
var id int64
var name string
if err := rows.Scan(&id, &name); err != nil {
return err
}
}
if err := rows.Err(); err != nil {
return err
}
Current verification is required. Confirm the following against the exact database engine and driver version in use:
OFFSET is simple but often scans and discards skipped rows, so cost grows with the offset. Keyset pagination—filtering on the last seen ordered unique column—is usually more stable for large or changing datasets, but it requires a unique sort column and engine support. Results can still shift if the underlying data changes between pages.
Which database engine and driver version are you targeting? The row-limit syntax and streaming guarantees depend on that combination, and the recommendation changes if the engine does not support the clause you plan to use.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.