Configuring pfSense REST API Pagination for Large Offset Queries
19K reputation · 28 Jul 2023, 21:26 UTC
Goal: Retrieve complete datasets from pfSense REST API endpoints (e.g., firewall rules) using the limit and offset parameters without losing records due to unexpected empty responses.
Uncertainty: In pfSense versions 2.6.0‑2.7.2, combining a very large offset (e.g., >100,000) with a small limit can cause the API to return an empty array instead of adjusting the offset or signaling an error, effectively hiding data beyond that threshold. It is unclear what offset range is safe for a given limit, whether a server‑side setting can modify this behavior, or if alternative pagination strategies (such as keyset pagination) are recommended.
Specific questions:
- What is the maximum offset value that guarantees a non‑empty page for a given limit size?
- Should client implementations treat an empty page as a signal to reduce the offset and retry, or is there a configurable API option to avoid this behavior?
- Is there a documented method to obtain accurate total counts or to use offset‑free pagination for large datasets in pfSense?
