Configuring pfSense REST API Pagination for Large Offset Queries

The short answer

There is no documented "safe maximum offset" for the pfSense REST API, and no server-side setting that changes deep-offset behavior. The reliable fix is to stop depending on large offsets: cap your page size, sort on a stable field, and treat an empty page as "end of data," not as a signal to retry with a different offset. If you genuinely need to walk hundreds of thousands of records, use filtered queries, snapshots, or an export job instead of offset paging.

What is confirmed vs. what is likely

Confirmed, general behavior: offset pagination requires the server to count or skip every row before the requested page. Cost grows with the offset even when the page itself is tiny, and results can skip or duplicate rows when the underlying data (firewall states, logs, leases) changes between requests.

Likely, but not verified for your install: the empty-array-at-large-offset behavior you describe in 2.6.0–2.7.2 is most plausibly an internal guard or an integer/limit boundary in the REST package you are using, not a tunable. Which parameters exist at all (limit, offset, total, sort fields) depends on whether your API is the built-in REST capability or a community package such as pfSense-api, and on the endpoint. Do not assume uniform behavior across endpoints.

What to do in this case

1. Identify exactly which REST implementation and version you are running, and read that endpoint's actual response: check for pagination metadata, documented max page sizes, and supported sort fields.
2. Pick the smallest page size that meets your throughput needs and always sort on a stable, unique field so page boundaries are deterministic.
3. Treat an empty page as termination. Do not retry with a reduced offset — if the data set were larger than the offset, the page would not be empty, and retrying just doubles the cost of the expensive query you already ran.
4. For large inventories (states, logs, aliases, DHCP leases), replace deep paging with server-side filters, time windows, or a periodic local snapshot you paginate yourself.
5. If consumers need bulk export, provide an asynchronous job or file download rather than letting clients walk to offsets in the tens of thousands.

Verifying the boundary yourself

You can find the practical offset ceiling empirically with a binary search against a known data set:

curl -s -u admin:YOURPASS \
  "https://pfsense.example/api/v1/firewall/rule?limit=100&offset=50000" \
  | jq 'length'

Double the offset until the response goes empty, then bisect. Compare latency at low and high offsets — a sharp slowdown before the empty page confirms the skip-cost explanation. Repeat while adding or removing a few rules to confirm whether pages shift. Note that any threshold you find this way is an observation about your version, not a guarantee; it can change across upgrades.

On totals and keyset pagination

If the endpoint exposes a total count, treat it as approximate for live tables — it can go stale mid-walk and may itself be expensive to compute. Keyset (cursor) pagination — "give me rows where id > last_seen" — avoids the offset problem entirely, but only if your endpoint supports filtering and sorting on an indexed field. Check the endpoint documentation for your package; if it is not supported there, the snapshot approach above is the pragmatic substitute.

One detail that would sharpen this answer: which REST implementation you are running (built-in vs. a specific community package and version), since parameter names and limits differ between them.