Implementing Server-Side Pagination in Django with Paginator

The Problem: Memory Exhaustion from Large QuerySets

Loading thousands of database records into a single HTML page causes two primary failures: browser crashes due to DOM overload and server-side memory exhaustion. When a Django view returns a full QuerySet, the application attempts to materialize every record into Python objects before rendering the template.

The solution is server-side pagination using django.core.paginator.Paginator. This utility allows you to slice a QuerySet at the database level using SQL LIMIT and OFFSET clauses, ensuring the server only processes a small subset of data per request.

Prerequisites

• A Django project with a defined Model and a corresponding QuerySet.
• A view function or class capable of handling GET requests.
• Django 3.2 or newer (though the Paginator API is stable across most versions).

Implementing the Pagination Logic

To implement pagination, you must capture the current page number from the request parameters, initialize the Paginator with your data source, and retrieve the specific page object.

Step 1: Configure the View

Run this logic within your view function. You will need to import Paginator from django.core.paginator.

from django.core.paginator import Paginator
from django.shortcuts import render
from .models import Product

def product_list(request):
    # 1. Fetch the full QuerySet (not yet evaluated)
    product_list = Product.objects.all()
    
    # 2. Initialize Paginator with the QuerySet and items per page
    paginator = Paginator(product_list, 25) 
    
    # 3. Get the page number from the URL (e.g., ?page=2)
    page_number = request.GET.get('page')
    
    # 4. Use get_page() to handle invalid page numbers or missing parameters
    # get_page() automatically handles EmptyPage and PageNotAnInteger exceptions
    page_obj = paginator.get_page(page_number)
    
    return render(request, 'products.html', {'page_obj': page_obj})

Step 2: Render Navigation in the Template

The page_obj contains the slice of data (object_list) and boolean flags to determine if navigation links should be displayed.

<!-- Loop through the items for the current page -->
<ul>
  {% for product in page_obj %}
    <li>{{ product.name }}</li>
  {% endfor %}
</ul>

<!-- Pagination Controls -->
<div class="pagination">
  {% if page_obj.has_previous %}
    <a href="?page={{ page_obj.previous_page_number }}">Previous</a>
  {% endif %}

  <span>Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}</span>

  {% if page_obj.has_next %}
    <a href="?page={{ page_obj.next_page_number }}">Next</a>
  {% endif %}
</div>

Performance Comparison: Paginator vs. Manual Slicing

Django's Paginator is efficient because it leverages QuerySet slicing. When you call page(), Django does not load the entire list; it applies a slice to the SQL query.

Method	SQL Behavior	Memory Impact	Use Case
.all()	SELECT * FROM table	High (Loads all)	Small datasets (<100 rows)
Paginator	SELECT ... LIMIT 25 OFFSET 50	Low (Loads 25)	Standard web views
.iterator()	Server-side cursor	Very Low	Massive exports/migrations

Diagnostics and Verification

To verify the pagination is working without loading the entire database into memory, you can use the Django shell.

Run in terminal: python manage.py shell

from django.core.paginator import Paginator
# Test with a range of 100 integers, 10 per page
items = range(100)
paginator = Paginator(items, 10)
page_1 = paginator.page(1)

# Check if the first page contains exactly 10 items
print(len(page_1.object_list)) # Expected: 10
# Check if the second page starts at the 11th item
page_2 = paginator.page(2)
print(page_2.object_list[0]) # Expected: 10 (0-indexed range)

Limitations and Risks

• Offset Performance: For extremely large tables (millions of rows), OFFSET becomes slow because the database must still scan through the skipped rows. In these cases, consider "keyset pagination" (filtering by the last ID seen).
• Direct URL Manipulation: If you use paginator.page() instead of paginator.get_page(), a user entering ?page=999999 in the URL will trigger an OutOfRangePage exception, resulting in a 500 error. Always use get_page() or wrap the call in a try-except block.
• Ordering: Pagination requires a consistent order. If your QuerySet is not ordered (e.g., missing ordering in Model Meta), the database may return items in a different order across pages, causing duplicate or missing items.