Schema-serialized pages or a streamed response for a large paginated Fastify endpoint?
0 reputation · 04 Sept 2026, 05:58 UTC
0 reputation · 04 Sept 2026, 05:58 UTC
A Fastify route returns pages from a table with millions of rows. Page size and cursor are bounded in the route's querystring schema with maximum constraints, so validation rejects unbounded requests before the handler runs. The decision is how the page payload leaves the server.
One approach uses a response schema so Fastify compiles a fast-json-stringify serializer for the repeated list payload. Serialization is fast but only fields declared in the schema are emitted, silently dropping undeclared properties. The alternative is reply.send with a Node.js Readable stream piped from the database driver, avoiding buffering the full result set in memory but losing schema serialization and per-payload validation, and complicating error handling once headers are flushed.
With a per-endpoint page cap of a few hundred rows, is the memory argument for streaming relevant, or is it only for unbounded exports? If schema serialization is kept, what is the accepted way to catch undeclared fields during development? Is exposing a total count via COUNT(*) alongside each page worth its cost on a large table?
31275 reputation · 04 Sept 2026, 09:11 UTC
Keep the response schema and Fastify's compiled serializer. With a page cap of a few hundred rows, the memory argument for streaming does not apply — it is a technique for unbounded exports, not bounded pages. Catch undeclared fields by letting Fastify's response validation fail loudly in development, and skip COUNT(*) unless a client genuinely needs it.
Streaming a Readable from the database driver exists to avoid buffering an unbounded result set. Your querystring schema already bounds the page at a few hundred rows, so the entire payload is, at most, a few hundred small objects — typically well under a few megabytes. You can confirm this directly:
const before = process.memoryUsage().heapUsed;
const rows = await db.query(/* bounded page query */);
console.log('heap delta MB:', (process.memoryUsage().heapUsed - before) / 1e6);If the delta is single-digit megabytes, buffering is a non-issue. In exchange for streaming you would lose the compiled serializer, lose response validation, and inherit the awkward problem of surfacing a mid-stream database error after headers have already been flushed — at that point the only honest option is aborting the connection, which clients experience as a truncated response rather than a clean 500.
The silent-drop behavior you describe happens because fast-json-stringify only emits declared properties. The accepted guardrail is Fastify's response validation, which runs before serialization: if the handler returns an object with properties not permitted by the schema (assuming the schema does not allow additional properties), validation fails and Fastify returns a 500 with a validation error. To make this visible instead of mysterious, wire an error handler in development:
app.setErrorHandler((err, req, reply) => {
if (err.validation) {
req.log.error({ validation: err.validation }, 'response schema mismatch');
}
reply.send(err);
});A fast unit test that injects a row with an extra property and asserts a 500 confirms the guardrail is active. Note the caveat: this only works if the payload goes through reply.send() with a response schema attached. Anything written via reply.raw bypasses both validation and serialization entirely, so undeclared fields will slip through silently there.
Usually not, on a millions-row table. A full COUNT(*) is roughly O(N) work per request (exact cost depends on the engine, indexes, and filtering), paid on every page fetch just to render a total. Practical alternatives, in rough order of preference:
totalCount, or only on the first page.Schema-serialized pages via reply.send(), response validation surfaced through a dev error handler, cursor-based pagination without a total count by default. Reserve streaming for a separate, explicitly unbounded export endpoint if one ever exists — and treat its post-headers error handling as a design problem of its own.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.