Implementing pagination for large datasets using Deno.serve()

Direct Answer

Return a Response whose body is a ReadableStream created with a pull callback that fetches one page at a time. Inside pull, read controller.desiredSize to pause production when the client falls behind, and use controller.enqueue(), controller.close(), and controller.error() to drive the HTTP response lifecycle. This keeps memory constant and respects backpressure without blocking the event loop.

Why This Works

Deno’s HTTP server implements the Web Fetch API: a handler returns a Response and the body can be any ReadableStream. When the body is a stream, Deno writes chunks directly to the socket using chunked transfer encoding. The stream’s pull function is called only when the internal queue has room, which is signaled by controller.desiredSize > 0. If the client reads slowly, desiredSize drops to zero or negative, and your pull simply waits (or yields) instead of enqueueing more data.

Minimal Implementation

const MAX_LIMIT = 1000;

Deno.serve({ port: 8080 }, async (req) => {
  const url = new URL(req.url);
  const limit = Math.min(parseInt(url.searchParams.get('limit') || '100'), MAX_LIMIT);
  const offset = parseInt(url.searchParams.get('offset') || '0');

  const stream = new ReadableStream({
    async pull(controller) {
      // 1. Check backpressure
      if (controller.desiredSize != null && controller.desiredSize <= 0) {
        await new Promise(r => setTimeout(r, 10)); // yield to event loop
        return;
      }

      // 2. Fetch next page from your data source
      const page = await fetchPage(offset, limit); // implement per your DB/file
      if (page.length === 0) {
        controller.close();
        return;
      }

      // 3. Encode and enqueue each record as NDJSON
      const encoder = new TextEncoder();
      for (const row of page) {
        controller.enqueue(encoder.encode(JSON.stringify(row) + '\n'));
      }

      offset += page.length;
    },
    cancel(reason) {
      console.log('Client disconnected:', reason);
    }
  });

  return new Response(stream, {
    headers: {
      'Content-Type': 'application/x-ndjson',
      'Cache-Control': 'no-cache'
    }
  });
});

Backpressure Controller Methods

• controller.desiredSize — read-only; positive means the internal queue can accept more bytes, zero or negative means pause. Treat null as unbounded (client not signaling).
• controller.enqueue(chunk) — adds a chunk (Uint8Array) to the queue. Call once per record or per small batch.
• controller.close() — signals end of stream; Deno sends the final chunk and closes the response.
• controller.error(err) — aborts the stream with an error; Deno emits a 5xx response if headers not yet sent.

Critical Practices

1. Respect desiredSize in every pull iteration. If you ignore it, the queue grows until memory exhausts.
2. Handle client abort. Listen to req.signal.addEventListener('abort', ...) and call controller.close() to free resources.
3. Serialize off the main thread for CPU-heavy data. Wrap JSON.stringify in a TransformStream or a Worker to avoid blocking the event loop.
4. Prefer keyset pagination. Offset/limit degrades on deep pages; use WHERE id > last_seen ORDER BY id LIMIT n for consistent performance.

Verification Steps

1. Run the server with deno run --allow-net server.ts.
2. Stream 10,000 objects: curl -N http://localhost:8080/data?limit=100 and observe chunked output.
3. Monitor RSS with deno run --allow-net --allow-sys server.ts and Deno.metrics(); memory should stay flat (~few MB).
4. Simulate slow client: curl --limit-rate 10k http://localhost:8080/data; server CPU should idle while desiredSize <= 0.
5. Abort mid-stream (Ctrl+C) and confirm no unhandled promise rejections.

One Missing Detail

What is the data source (PostgreSQL, Deno KV, SQLite, flat file, generated)? The fetchPage implementation changes significantly: cursor-based drivers (PostgreSQL portal, Deno KV) yield rows natively, while files need a streaming parser. Knowing the source lets me show the exact pull logic that avoids any intermediate array allocation.