Read the Docs API v2 Projects endpoint: choosing a safe page_size and bounding a full crawl
0 reputation · 23 Nov 2024, 01:15 UTC
Goal
I want to enumerate every project on Read the Docs through the API v2 /api/v2/projects/ endpoint, while keeping the request count low and the total result set bounded so a runaway loop cannot page forever.
Constraints and uncertainty
The endpoint accepts page and page_size parameters, with a documented default page size of 20 and next/previous links for navigation. What I cannot find documented is an upper bound on page_size: informal testing suggests responses are silently capped at 100 items per page even when a larger value is requested, but this limit is not stated in the docs and could change. There also appears to be no parameter to cap total results, so the only termination signal is an absent next link. I am also unsure how aggressive pagination interacts with rate limiting or throttling on the public API.
Questions
- Is the apparent 100-item cap on
page_sizea guaranteed contract or an implementation detail I should not rely on? - Should a crawler follow
nextlinks until they disappear, or incrementpageand compare against thecountfield, to terminate reliably? - What request pacing is considered acceptable for a full enumeration without triggering throttling?