Cross-version search returns incomplete results after version push
0 reputation · 12 Feb 2022, 21:04 UTC
0 reputation · 12 Feb 2022, 21:04 UTC
When querying across multiple documentation versions using the Read the Docs search API, results for older versions occasionally omit documents that are known to exist in those versions' built output. The issue appears after new versions are pushed and indexed, suggesting a synchronization gap between version-specific Elasticsearch indexes.
The project uses the Elasticsearch backend configured via ELASTICSEARCH_URL in .readthedocs.yaml. Each version push triggers a reindex of that version's content, while prior versions retain their own indexes for backward compatibility. Cross-version queries rely on querying multiple indexes simultaneously.
Documentation indicates that indexes can drift out of sync, but the exact conditions — whether caused by partial indexing failures, mapping changes, or TTL-based cleanup — are not specified. The Whoosh fallback backend is not in use.
Achieve consistent cross-version search coverage so that a query with version=latest and version=stable returns all matching documents from both indexes without manual reindexing.
1. What causes an index to become stale? When a new version is built, Read the Docs reindexes only that version. If the reindexing process exits before all documents are indexed—due to timeouts, resource limits, or unexpected errors—the resulting index may be incomplete. Mapping changes introduced in newer builds can also make older indexes reject documents that no longer match the updated schema. Finally, the platform’s retention policy may delete old indexes while a new one is still being built, creating a race condition.
2. Is there a way to see index health? The platform shows index status in the project’s admin dashboard. While a public API endpoint for index health is not documented, the UI typically lists each version’s index as green, yellow, or red. If you have API access, you can try the hidden endpoint /api/v3/projects/slug/indexes/, but its availability is not guaranteed.
3. Can a project‑wide reindex be run safely? Yes. In the project settings you can trigger a “Rebuild Index” operation. It runs asynchronously and does not block the serving of the HTML pages, though it may temporarily increase search latency.
latest and stable are marked as green. If you have API access, a quick call like:
curl -s https://readthedocs.org/api/v3/projects/<slug>/indexes/ | jq
Completed and no error logs appear in the build history.?version=stable) and compare the URLs returned with the files present in the stable build directory.To determine whether a mapping mismatch is the culprit, please provide the latest mapping snapshot or any build‑time indexing error logs that mention “mapping conflict” or similar terms.
Use comments to ask for clarification. Post a solution as an answer.
No question comments on this page.