Read the Docs API v2 Projects endpoint: choosing a safe page_size and bounding a full crawl

Answer

1. Page size: Treat the 100‑item cap as an implementation detail. Use page_size=100 to stay within the documented safe range and avoid surprises if the server silently truncates larger requests.

2. Termination: Follow the next link until it is missing. Relying on the count field is fragile because the API could change its pagination limits or reorder items.

3. Rate‑limit pacing: Send at most one request per second (or a burst of up to 10 followed by a pause) to stay well below the typical 60 req/min threshold for unauthenticated traffic. If you receive a 429 response, back off exponentially with jitter.

Explanation

The v2 Projects endpoint defaults to 20 items per page, but empirical tests show a hard cap of 100 items per page even if you request more. This limit is not part of the public contract, so a crawler should not rely on it existing forever. Using page_size=100 gives the maximum stable payload while keeping the request size reasonable.

Pagination is driven by the next URL in the response. This URL is a full absolute link; reconstructing query parameters is unnecessary and error‑prone. Incrementing page and comparing against count can miss pages if the API changes its page size or if count is rounded. The safest termination condition is the absence of a next field.

Public APIs typically enforce around 60 requests per minute per IP for unauthenticated traffic. A conservative pacing of one request per second (or a short burst followed by a pause) keeps the rate well below that threshold. If you hit a 429, implement exponential backoff with a small random jitter before retrying.

To verify your crawler:

• Send a request with page_size=200 and confirm the results array never exceeds 100 items.
• Iterate using next and check that the set of project IDs is contiguous and contains no duplicates.
• Log any 429 responses and adjust the delay if needed.

Practical example (Python‑style pseudocode)

import time, requests
url = 'https://readthedocs.org/api/v2/projects/'
params = {'page_size': 100}
while url:
    r = requests.get(url, params=params)
    if r.status_code == 429:
        time.sleep(2)  # simple back‑off
        continue
    data = r.json()
    for project in data['results']:
        process(project)
    url = data.get('next')
    params = None  # next is absolute
    time.sleep(1)  # pacing

Replace process(project) with whatever logic you need. The loop will terminate once next is None, guaranteeing a complete crawl without endless paging.

Ask for clarification

If you have access to rate‑limit headers (e.g., X‑RateLimit‑Remaining), let me know; that can fine‑tune the pacing further.