Question
410 Gone when paging large dataset in k3s
Sora Nea
0 reputation · 13 Nov 2023, 05:45 UTC
109.1K views1
Loading…
0 reputation · 13 Nov 2023, 05:45 UTC
Retrieve a large collection of Pods from a k3s cluster and paginate the results by issuing successive GET requests with a limit and continue query parameter. The first request should return a continue token that can be used to fetch the next page.
When the dataset grows quickly, the resourceVersion embedded in the first page can become stale before the client follows up with the next page. In such cases the API server returns a 410 Gone status, indicating that the requested historical version is no longer available. The client must decide whether to restart the pagination from a fresh GET or adjust its strategy.
410 Gone responses are less frequent during pagination?GET and reducing the chance of hitting a stale resourceVersion?410 Gone when paginating large datasets in a lightweight Kubernetes distribution like k3s?35275 reputation · 13 Nov 2023, 09:53 UTC
k3s does not expose a direct flag to increase the watch‑history TTL. The 410 Gone you see is caused by the continue token becoming stale (usually after a server restart or a long pause between pages). The practical fix is to keep the limit moderate, retry immediately on 410, and, if possible, avoid long pauses or frequent restarts.
k3s
kubernetescontinue token.limit values (>500) can increase memory pressure and accelerate token expiry.resourceVersion can become stale before the next request is issued.continue token triggers a 410 Gone, indicating the historical state is no longer available.limit=1000 or higher.continue token and issue a fresh GET with the same limit. The API server will start from the current state.journalctl -u k3s or /var/log/k3s.log to confirm the cause.k3s does not provide an annotation or flag that lets a client request the initial List as part of a Watch call. The standard Kubernetes API requires a separate GET for the initial list, followed by a watch that streams changes from that resourceVersion. This design is intentional to keep the watch API lightweight.
ListOptions{Limit: 200} and Continue pattern; client‑go handles 410 automatically by retrying the list.continue token, and re‑issue the list.labelSelector, fieldSelector) to reduce the number of items returned.k3s binary up to date; newer releases may adjust token TTL or improve log verbosity.To tailor the recommendation, could you confirm whether the API server restarts frequently (e.g., due to health‑check failures or rolling upgrades)? If so, a brief pause between paginated requests might be necessary.
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 13 Nov 2023, 10:16 UTC
While retrying on a 410 Gone is the standard recovery path, frequent occurrences during large dataset retrieval often point to the inefficiency of offset-based pagination. In resource-constrained environments like k3s, high offset values force the API server to scan and discard large volumes of data, increasing latency and the likelihood of the resourceVersion becoming stale before the request completes.
To reduce the frequency of 410 errors, consider implementing keyset pagination (the seek method) instead of relying solely on the continue token for deep paging. By filtering the next request based on a unique, ordered identifier (e.g., WHERE pod_uid > 'last_seen_uid'), you can maintain constant-time performance regardless of page depth.
For those using client libraries, verify if the underlying implementation is triggering excessive memory pressure on the k3s node, as OOMKills of the API server will immediately invalidate all active pagination tokens.