Custom Pagination in Django REST Framework: Scaling List Endpoints for Large Datasets

The Problem – Large List Endpoints Overwhelming Clients

When an API returns a long list of objects, the default PageNumberPagination in Django REST Framework (DRF) sends 100 items per page. For many applications—especially those built with React, Vue, or mobile clients—this number is too small or too rigid. Clients may need more items per request to reduce round‑trips, but the server must also guard against excessive payloads that can strain memory and bandwidth.

The Thesis – A Flexible, Safe Custom Pagination Class

By extending PageNumberPagination we can expose a page_size query parameter that lets clients request up to a configurable maximum. The server enforces a hard cap, preventing abuse, while still offering flexibility. The class can also be tweaked to prefetch related objects or cache page results, further improving performance on read‑heavy APIs.

Exposing a page_size Query Parameter

DRF’s base pagination class hides the page_size parameter. Adding it to the custom class allows clients to override the default page size:

from rest_framework.pagination import PageNumberPagination

class FlexiblePageNumberPagination(PageNumberPagination):
    page_size_query_param = "page_size"
    # default_page_size is optional; if omitted, DRF uses 10

Enforcing a Hard Max Size

To protect the server, define a max_page_size attribute. Requests that exceed this limit will be capped automatically:

class FlexiblePageNumberPagination(PageNumberPagination):
    page_size_query_param = "page_size"
    max_page_size = 200  # maximum items per page

Building the Custom Class

Below is a minimal implementation that also prefetches a related model to reduce database hits. Replace RelatedModel with your actual relation.

from rest_framework.pagination import PageNumberPagination
from django.db.models import Prefetch

class PrefetchingPagination(PageNumberPagination):
    page_size_query_param = "page_size"
    max_page_size = 200
    default_page_size = 50

    def paginate_queryset(self, queryset, request, view=None):
        # Example: prefetch a related model named "author"
        queryset = queryset.prefetch_related(Prefetch('author', queryset=Author.objects.only('id', 'name')))
        return super().paginate_queryset(queryset, request, view)

Hooking it into the API

Configure the pagination class globally in settings.py or per view. Global configuration keeps the API consistent:

# settings.py
REST_FRAMEWORK = {
    "DEFAULT_PAGINATION_CLASS": "myapp.pagination.PrefetchingPagination",
    "PAGE_SIZE": 50,
}

Alternatively, set it on a specific ViewSet:

class ArticleViewSet(viewsets.ModelViewSet):
    queryset = Article.objects.all()
    serializer_class = ArticleSerializer
    pagination_class = PrefetchingPagination

Performance Gains – One Query per Page

With custom pagination, each request issues a single SELECT with LIMIT and OFFSET. Even with 200 items per page, the database only retrieves the required slice, keeping memory usage low.

To verify, you can:

• Run a benchmark: create 500,000 dummy articles, then request page 1 with ?page_size=200 and compare response time to the default 100.
• Check the Django logs or use the Django Debug Toolbar to confirm a single query with LIMIT and OFFSET per request.

Trade‑offs & Limitations

• Abuse Risk: Exposing page_size invites DoS attempts. The max_page_size guard mitigates this; consider adding rate limiting if needed.
• Prefetch Overhead: Prefetching unrelated data can increase query time. Profile your queries to ensure the benefit outweighs the cost.
• Caching Complexity: Caching page results is useful but requires cache invalidation logic. Use Django’s cache_page decorator or a custom cache backend if your data changes frequently.

Practical Verification Checklist

1. Generate a test dataset (~500k records).
2. Enable the custom pagination class.
3. Send a request: GET /api/articles/?page=1&page_size=200.
4. Measure response time and compare to the default 100.
5. Inspect the SQL query in logs: it should contain LIMIT 200 OFFSET 0 and SELECT only the requested fields.
6. Confirm that the response JSON includes count, next, previous, and results keys.

Takeaway

Custom pagination in DRF is a lightweight, powerful tool for scaling list endpoints. By exposing a page_size parameter, enforcing a hard cap, and optionally prefetching related data, you can deliver predictable payloads that keep both server and client performance in check. Start by adding the PrefetchingPagination class to your project, run the verification steps above, and adjust max_page_size as your workload demands.