Architecting Pagination in Django REST Framework: Boundaries, Failure Modes, and Scaling Limits

The Problem: Unbounded Responses and Shifting Result Windows

When you expose a list endpoint in Django REST Framework (DRF), the framework does not paginate by default. A single GET /api/articles/ request can return every row in the table. That is acceptable for a local seed script, but in production it creates two immediate problems: clients can force unbounded response sizes, and database queries that scan large tables without limits will eventually time out. The useful takeaway is that pagination is a server-side contract, not a client preference. You choose the paginator, enforce its limits, and design for the failure modes of the strategy you pick.

Requirements for a Pagination Layer

Before selecting a class, define what the API must guarantee:

• Bounded payloads: No client-supplied parameter should be able to request an arbitrarily large page.
• Deterministic ordering: Every page must be sorted by a column (or set of columns) that produces a total order. Without a unique tie-breaker, the same row can appear on two pages or vanish entirely.
• Predictable client contract: The response shape, parameter names, and error behavior should be stable across releases.
• Resource protection: The database must not be forced into full-table scans or expensive sorts by query-string values.

The Smallest Suitable Design: Capped PageNumberPagination

For a new API backed by a small, stable dataset, the smallest suitable design is PageNumberPagination with a capped page size and explicit ordering. It requires no schema changes, maps cleanly to client expectations, and is easy to test.

Start with a custom pagination class in myapp/pagination.py:

from rest_framework.pagination import PageNumberPagination

class StandardResultsSetPagination(PageNumberPagination):
    page_size = 25
    page_size_query_param = 'page_size'
    max_page_size = 100

Then apply it globally in settings.py or per-view:

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PAGINATION_CLASS': 'myapp.pagination.StandardResultsSetPagination',
    'PAGE_SIZE': 25,
}

# myapp/views.py
from rest_framework import generics
from .models import Article

class ArticleListView(generics.ListAPIView):
    queryset = Article.objects.all().order_by('published_at', 'id')
    pagination_class = StandardResultsSetPagination

The order_by('published_at', 'id') clause is not optional. If published_at is not unique, adding the primary key as a tie-breaker prevents duplicate rows across pages. The max_page_size = 100 line is the critical server-side guard: even if a client sends ?page_size=10000, the response is capped.

Trust and Data Boundaries

The query string is an untrusted input boundary. Clients control page, page_size, offset, and limit. Any value they supply must be validated and bounded by the server. In particular:

• Enforce max_page_size: Without it, a client can request a page large enough to exhaust memory or hold a database connection.
• Whitelist ordering fields: If you allow clients to sort via a query parameter, restrict the choices to indexed columns. Accepting arbitrary field names can trigger unindexed sorts or expose internal columns.
• Treat cursors as opaque: CursorPagination encodes position in a base64 string. It is not encrypted. Do not embed sensitive data in the ordering fields, and do not treat the cursor as a secure session token.

Operational Checks

Configuration alone does not prove the design works. Run these checks before and after deployment:

1. Index usage: In the Django shell or a database client, run EXPLAIN on the queryset that backs the list view. Confirm the plan uses an index on the ordering column rather than a sequential scan. For example: EXPLAIN SELECT * FROM myapp_article ORDER BY published_at, id LIMIT 25 OFFSET 5000;
2. Latency at depth: Measure response time for a high page number, such as ?page=2000. If latency grows linearly with the offset, the database is scanning skipped rows.
3. Cache behavior: If you use Django's cache framework or a CDN, verify that repeated identical requests hit the cache and that cache keys include the pagination parameters.

Failure Modes

Paginator	Best for	Client controls	Consistency under writes	Count included
PageNumberPagination	Small, stable datasets	page, page_size	Shifts when rows are inserted or deleted	Yes
LimitOffsetPagination	Simple offset control	limit, offset	Shifts when rows are inserted or deleted	Yes
CursorPagination	Large or high-write datasets	cursor (opaque)	Stable keyset position	No

Offset-based pagination (PageNumberPagination and LimitOffsetPagination) has a well-known consistency flaw: if a new row is inserted before the current window while a client is paging, subsequent pages shift and the client may miss or repeat records. Cursor pagination avoids this by encoding a keyset position, but it introduces its own failure mode: if the row referenced by a cursor is deleted, DRF may return a 404 unless the view handles the missing position gracefully. Additionally, CursorPagination does not include a total count by default; if your UI requires "page X of Y," you must either compute the count separately or accept the missing field.

When to Move Beyond Offset Pagination

Change the design when the dataset or traffic pattern makes offset scanning unacceptable:

• Datasets exceeding millions of rows: Deep offsets force the database to read and discard rows. CursorPagination uses a WHERE clause on the ordering column instead of OFFSET, keeping query cost roughly constant.
• High write concurrency: If inserts and deletes are frequent, offset pages become unreliable. Cursor pagination anchors to a unique ordering value, so the window does not shift.
• Bidirectional traversal: Clients that need to page backward efficiently benefit from cursors, which encode both forward and reverse positions.

To switch to CursorPagination, the ordering field must be unique and immutable for the life of the cursor. The primary key is the safest choice:

# myapp/pagination.py
from rest_framework.pagination import CursorPagination

class ArticleCursorPagination(CursorPagination):
    page_size = 25
    ordering = 'id'

If you must order by a non-unique field such as created_at, include the primary key as a secondary sort: ordering = ('created_at', 'id'). Without the tie-breaker, rows with identical timestamps can be skipped or duplicated.

Verification and Limitations

Write tests that assert the boundary behavior, not just the happy path:

• Cap enforcement: Request ?page_size=10000 and assert the response contains at most max_page_size items.
• Offset shift: In an integration test, request page 1, insert a new row that sorts before the current window, request page 2, and observe whether the expected records appear. This documents the consistency limitation of offset pagination.
• Version check: Confirm the installed DRF version supports the settings you use. The DEFAULT_PAGINATION_CLASS and PAGE_SIZE keys have been available for many releases, but always check the documentation for your installed version.

Changing the default pagination class or ordering in a released API is a breaking change. Existing clients may send parameters that no longer exist or rely on a response shape that has changed. Version the API or deprecate the old behavior with a documented migration window before rolling out the new paginator.

Finally, built-in DRF paginators do not solve every problem. They do not provide efficient total counts for cursor pagination, they do not encrypt cursors, and they do not automatically create database indexes. The paginator is a contract layer; the database schema and query patterns underneath it still determine whether the API scales.