Managing Large API Datasets with Django REST Framework PageNumberPagination

The Problem: Response Bloat and Database Latency

Returning thousands of records in a single JSON array causes high memory usage on the server, slow transmission times over the network, and browser crashes on the client side. Without pagination, a simple GET /api/users/ request can scale from a few milliseconds to several seconds as the database grows.

The immediate solution is PageNumberPagination, which splits the dataset into discrete pages. This ensures that the API response remains constant in size regardless of the total record count.

Prerequisites

• Django REST Framework (DRF) installed and configured.
• A model with a significant number of records (e.g., 100+).
• A ViewSet or GenericAPIView implementing a list method.

Global Pagination Configuration

Setting pagination globally is the most efficient way to ensure consistency across your API. This configuration applies to all views that inherit from GenericAPIView or ViewSet.

Add the following to your settings.py file:

REST_FRAMEWORK = { 
    'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.PageNumberPagination', 
    'PAGE_SIZE': 20 
}

Key Parameters:

• DEFAULT_PAGINATION_CLASS: Specifies the logic used to slice the queryset.
• PAGE_SIZE: The maximum number of items returned per request.

Implementing Custom Pagination per Endpoint

Global settings are often too rigid. For example, a "Search" endpoint might require 50 results per page, while a "User Profile" list only needs 10. To achieve this, create a custom pagination class by subclassing PageNumberPagination.

In a new file, such as pagination.py:

from rest_framework.pagination import PageNumberPagination

class StandardResultsSetPagination(PageNumberPagination):
    page_size = 50
    page_query_param = 'p' # Changes the URL parameter from ?page= to ?p=

Then, apply this class to a specific view in views.py:

from rest_framework import generics
from .models import Product
from .serializers import ProductSerializer
from .pagination import StandardResultsSetPagination

class ProductListView(generics.ListAPIView):
    queryset = Product.objects.all()
    serializer_class = ProductSerializer
    pagination_class = StandardResultsSetPagination

Response Structure Comparison

Pagination changes the shape of your API response. Instead of a raw list [...], DRF wraps the data in an object to provide navigation metadata.

Unpaginated Response	Paginated Response
[ {"id": 1, ...}, {"id": 2, ...} ]	{ "count": 100, "next": "/api/p=2", "previous": null, "results": [ {"id": 1, ...} ] }

Verification and Diagnostics

To verify the implementation, run your server and perform the following checks using a tool like cURL or Postman:

1. Check Structure: Request GET /api/products/. Ensure the response is an object containing the results key, not a top-level array.
2. Test Navigation: Click the URL provided in the next field. Verify that the results array contains a different set of records.
3. Parameter Validation: Request GET /api/products/?page=2 (or ?p=2 if customized). Confirm that the response returns the second set of records.

Performance Limitations

PageNumberPagination uses SQL OFFSET and LIMIT clauses. As the page number increases (e.g., page 10,000), the database must still scan through all preceding rows before returning the requested slice, which leads to performance degradation.

Decision Matrix:

• Use PageNumberPagination: When you need a simple "Page 1, 2, 3" UI and datasets are moderately sized (thousands of rows).
• Use CursorPagination: For extremely large datasets (millions of rows) or infinite-scroll UIs, as it uses a pointer to the last record rather than an offset.

Rollback Procedure

To remove pagination and return to returning full lists, remove the DEFAULT_PAGINATION_CLASS from settings.py and delete the pagination_class attribute from individual views.