Enumerating Complete NetBox REST API Result Sets with Limit/Offset Pagination

Desired Outcome: A Complete, Deduplicated Record Set

When you need to export every device, IP address, or circuit from NetBox, the REST API will not return the full table in a single response. The default page size is 50 records, and the maximum is set per deployment rather than being a universal constant. A client that stops after the first page silently loses data. The reliable approach is to treat every list endpoint as a paginated stream: request a bounded page, accumulate the results array, advance the offset, and verify that the number of unique objects matches the count field in the response envelope.

Prerequisites

• A NetBox instance reachable over HTTPS and an API token created for a user with read permissions on the target objects. Tokens are passed in the Authorization: Token <key> header and inherit the creating user’s object permissions.
• An HTTP client such as curl or Python’s requests library.
• The base URL of your instance, for example https://netbox.example.com, and the specific API path you intend to enumerate, such as /api/dcim/devices/.
• Awareness of the deployed NetBox major version. The REST API is versioned, and response schemas can evolve between releases, so pin your client to the version in use and review release notes before upgrading.

Step 1: Verify the Pagination Envelope

Before writing a loop, confirm that the endpoint behaves as expected. Issue a minimal request with limit=1 and inspect the JSON keys.

Where to run: any shell or HTTP client with network access to the NetBox instance.

curl -s -H "Authorization: Token <NETBOX_API_TOKEN>" \
  "https://<NETBOX_HOST>/api/dcim/devices/?limit=1&offset=0"

Expected check: an HTTP 200 response whose JSON body contains the keys count, next, previous, and results. The count value reflects the total matching records at the moment of that first request. If you receive a 401 or 403, treat it as an authentication or authorization failure, not as an empty result set.

Step 2: Page Through the Result Set

NetBox uses Django REST Framework LimitOffsetPagination. You pass limit and offset as query parameters, and the server returns a page plus navigation metadata. A robust loop requests ?limit=<n>&offset=<m>, accumulates results, then advances offset by the number of records returned until next is null or fewer than limit records arrive.

The following Python sketch illustrates the pattern. Replace the placeholders with your actual host and token.

import requests

base = "https://<NETBOX_HOST>/api/dcim/devices/"
headers = {"Authorization": "Token <NETBOX_API_TOKEN>"}
limit = 100
offset = 0
seen_ids = set()
records = []

while True:
    resp = requests.get(
        base,
        headers=headers,
        params={"limit": limit, "offset": offset},
        timeout=30,
    )
    resp.raise_for_status()
    payload = resp.json()

    for item in payload["results"]:
        obj_id = item["id"]
        if obj_id not in seen_ids:
            seen_ids.add(obj_id)
            records.append(item)

    if payload["next"] is None:
        break

    offset += len(payload["results"])

Relevant risk: do not hard-code limit=1000. The maximum page size is configurable by the administrator, and requesting a value above the configured maximum returns a validation error rather than silently truncating the page. Start with a conservative value such as 50 or 100 and adjust only after confirming the deployment’s limits.

Step 3: Guard Against Concurrent Changes

The count field reflects the moment of the first request. If other users or automation create or delete objects while your sweep is running, later pages can shift. You may see duplicate records or skip entries entirely. Deduplicate by object id as you accumulate, and if the final unique count does not match the initial count, re-query the small gap rather than trusting a single pass when data is churning.

For example, if the first page reported count: 842 and you collected 838 unique IDs, do not assume the missing four are inaccessible. Re-run the sweep with a narrower filter or a smaller limit to confirm whether they are new, deleted, or permissions-scoped.

Step 4: Tune for Large Tables

Offset pagination asks the database to scan and discard skipped rows. On very large tables, a high offset becomes progressively slower and can exhaust worker memory or request timeouts. Instead of one unbounded sweep, narrow the query with filters such as site, role, or status, and process each subset separately. This keeps each page’s database work bounded and makes failures easier to isolate.

Strategy	When to use	Trade-off
Single sweep, limit=100	Small to medium tables, stable data	Simple, but deep offsets slow down
Filtered subsets by site or role	Large tables, distributed ownership	More requests, but each is fast and restartable
Smaller limit for validation	Confirming boundary handling	More round trips, useful as a correctness check

Expected Checks

1. Shape check: the first response includes count, next, previous, and results.
2. Scope check: compare the count from the API against the object total shown in the NetBox UI for the same filter to confirm you are querying the intended set.
3. Completeness check: after a full sweep, assert that the number of unique id values collected equals count and that no id repeats.
4. Boundary check: re-run the sweep with a smaller limit (for example 25) and confirm the same unique-id set is produced. Matching sets across page sizes prove that your offset arithmetic and termination condition are correct.
5. Performance check: measure response time for a large offset before relying on deep pagination in scheduled jobs. If latency climbs sharply, switch to filtered subsets.

Recovery Options and Limitations

This operation is read-only, so there is no state to roll back. Recovery focuses on diagnosing incomplete or failed runs.

• 401 or 403 responses: the token is missing, revoked, or lacks permissions. Regenerate or re-scope the token rather than treating the response as an empty page.
• 400 validation errors: usually an excessive limit. Reduce the page size and retry.
• Timeouts or memory pressure: abandon the unbounded sweep, add filters, and process subsets. Persist the last successful offset per subset so a retry resumes rather than restarting.
• Mismatched counts: if unique IDs do not equal count, re-query the affected range with a tighter filter or smaller page size. Concurrent writes are the most common cause.

Limitations: offset pagination is not a transactional snapshot. The count is advisory, and deep offsets impose real database cost. NetBox’s REST API is versioned; pin your client to the deployed major version and validate response schemas after any NetBox upgrade. When in doubt, verify against the UI and re-run with a smaller page size before trusting an automated export.