Choosing Django REST Framework Pagination for Large Querysets

Problem Statement & Key Takeaway

REST APIs that return large querysets risk excessive memory usage, slow network transfers, and database overload. Pagination separates the client’s request for a page from the server’s need to load all rows, delivering only a slice of data per response. The takeaway: use Django REST Framework’s built‑in pagination classes, cap request‑controlled page sizes, and pick the strategy that matches the UI pattern and data consistency needs.

Requirements

• Serve querysets that can exceed 10 000 rows in production.
• Expose a predictable total count to clients without forcing a full‑count query on every request.
• Prevent request amplification attacks where a malicious client requests an arbitrarily large page.
• Support infinite‑scroll UIs that need stable ordering across concurrent writes.
• Maintain a single source of truth for pagination configuration to avoid per‑view drift.

Minimal Suitable Design

The simplest design leverages the global REST_FRAMEWORK setting. Choose a pagination class that balances performance and UX:

# settings.py
REST_FRAMEWORK = {
    # Global pagination for all API views
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination',
    # Default maximum size a client can request
    'PAGE_SIZE': 100,
    'PAGE_SIZE_QUERY_PARAM': 'page_size',
    'MAX_PAGE_SIZE': 200,
}

With this setup:

• Every view that returns a QuerySet automatically paginates.
• The client can override the page size up to 200 via ?page_size=150.
• Requests for ?page=1 return the first 100 items by default.

For infinite scroll, replace the default class with CursorPagination and specify a stable ordering field:

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.CursorPagination',
    'PAGE_SIZE': 50,
    'ORDERING_FIELD': 'id',  # must be unique and indexed
}

Cursor pagination eliminates record duplication or omission when new rows are inserted or existing rows are deleted between requests.

Trust & Data Boundaries

• Client‑controlled page_size: Only allow values up to MAX_PAGE_SIZE. Larger values should be clamped to the maximum to avoid memory spikes.
• Ordering field for cursor pagination must be unique and stable. Using a non‑unique field (e.g., created_at) can produce gaps or duplicates.
• Do not expose raw database identifiers or internal pagination tokens to the client unless they are opaque and signed (DRF’s cursor tokens are base64‑encoded and signed by default).
• When using filters or ordering that depend on user input, validate and sanitize those parameters before they reach the pagination layer.

Operational Checks

1. Response Size – Verify that the Content-Length header matches the expected page_size × approximate size of each item. Run from a terminal with network access to the API host:

   curl -s -D - "http://localhost:8000/api/items/?page=1" | grep -i Content-Length
   

   Check that the value is reasonable (e.g., not exceeding a few hundred kilobytes for a page of 100 items). No special permissions are needed beyond read access to the endpoint.
2. Link Header – PageNumberPagination includes a Link header with rel=\"next\" and rel=\"prev\". Ensure it is present and correctly formatted:

   curl -s -D - "http://localhost:8000/api/items/?page=1" | grep -i Link
   

   If missing, verify that pagination is enabled in settings and that the view returns a QuerySet.
3. Database Query Count – Use Django Debug Toolbar or a custom middleware to confirm that only LIMIT and OFFSET (or cursor‑based) queries are executed, not a full SELECT COUNT(*) for every request unless count=True is required.
4. Cursor Validity – For cursor pagination, perform a two‑step request: first ?cursor=<empty> to get the initial cursor, then ?cursor=<token> for the next page. Verify that the returned slice is contiguous and that the total count field remains accurate. No special permissions beyond read access are needed.

Failure Modes

• Large page_size abuse: A malicious client requests ?page_size=1000000. The server clamps to MAX_PAGE_SIZE, but the database still performs a large OFFSET query, which can be slow. Mitigation: enforce a hard limit and log oversized requests.
• Cursor token tampering: If the client decodes the base64 cursor and manipulates it, the signature validation fails and the API returns a 400 error. This is safe but can be used for denial‑of‑service if abused. Mitigation: rate‑limit cursor requests.
• Ordering changes: If the ordering field for cursor pagination changes (e.g., from id to created_at), previously issued cursors become invalid. Clients will get errors or duplicate data. Mitigation: version the API or invalidate cursors on schema changes.
• Queryset size grows beyond expectations: Even with pagination, a SELECT COUNT(*) can become expensive. Enable count=True only when necessary or use approximate counts.

When to Pivot the Design

• High concurrency writes cause frequent page shifts. Cursor pagination is preferable over offset pagination because it preserves ordering consistency.
• Clients need total counts for pagination controls. PageNumberPagination automatically includes count, but cursor pagination can expose it with count=True at the cost of a full count query.
• Data is highly dynamic (e.g., real‑time feeds). Use LimitOffsetPagination with a small offset and limit to reduce the impact of shifting rows.
• API is public and load‑tested and you observe a high rate of request amplification. Tighten MAX_PAGE_SIZE or switch to a custom rate‑limiting middleware.
• You need to expose custom metadata (e.g., total pages, estimated processing time). Create a subclass of PageNumberPagination that adds these fields to the response data.

Concrete Example: Custom Pagination Class

Below is a minimal custom class that adds a total_pages field while retaining the default behavior:

# pagination.py
from rest_framework.pagination import PageNumberPagination
import math

class PageNumberWithMetadata(PageNumberPagination):
    def get_paginated_response(self, data):
        total_pages = math.ceil(self.page.paginator.count / self.page_size)
        return super().get_paginated_response(
            {
                'count': self.page.paginator.count,
                'total_pages': total_pages,
                'next': self.get_next_link(),
                'previous': self.get_previous_link(),
                'results': data,
            }
        )

Configure it globally:

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'myapp.pagination.PageNumberWithMetadata',
    'PAGE_SIZE': 100,
    'MAX_PAGE_SIZE': 200,
}

Verify that the response includes the new total_pages field by inspecting the JSON body with curl -s and jq (or any JSON viewer). No special permissions are required beyond read access to the endpoint.

Conclusion

Pagination is a foundational design choice for any API that exposes large datasets. By setting a global, capped pagination strategy, validating ordering fields, and monitoring response characteristics, you can keep the API responsive, secure, and maintainable. When the data model or client requirements evolve, revisit the pagination strategy with the failure modes and operational checks outlined above to ensure continued performance and correctness.