Kubeflow Pipelines experiment run pagination default pageSize ordering gap

Direct Answer

The pageToken in Kubeflow Pipelines ListRuns does not enforce ordering by itself. Ordering is controlled exclusively by the sort_by parameter (e.g., created_at desc). Without an explicit sort_by, the backend returns results in an undefined order, and concurrent run creation or deletion during traversal can produce duplicates or gaps across pages. The token merely encodes the server-side query position for that specific sort_by + filter combination at the moment the token was issued.

Confirmed Behavior

• Token lifecycle: A pageToken is an opaque continuation token bound to the exact query snapshot (sort order, filter, page size) that produced it. It expires if the backend state changes significantly (e.g., index rebuild, cluster restart) or after an implementation-dependent TTL. Long pauses between requests risk token invalidation.
• Result consistency: With a fixed sort_by, consecutive calls using the returned pageToken yield a stable, non-overlapping sequence for that snapshot. New runs inserted before the current cursor position in the sort order will not appear in subsequent pages; runs inserted after will appear. Deleted runs simply vanish.
• No cursor alternative: The API does not expose a stable, time-travel cursor (e.g., a run ID or timestamp boundary) that survives deployments or schema changes.

Cross-Project Pagination Comparison

There is no unified pagination primitive across Kubeflow. Katib has migrated toward token-based pagination in recent releases, but the token format and lifecycle differ from KFP. Notebook file browsing generally lacks server-side pagination entirely.

Deployment-Specific pageSize Maximum

No universal maximum exists in the KFP API specification. The effective limit is determined by:

• API gateway / ingress request-body size limits (often 4–16 MB)
• Backend gRPC message size limits (default ~4 MB in many deployments)
• Database query row-limit guards (some operators set hard caps, e.g., 1000 or 5000)

Assuming a page size larger than ~1000 without verification is unsafe. Silent truncation or 413 Payload Too Large / ResourceExhausted errors are common when the limit is exceeded.

Verification Steps for Your Deployment

1. Fetch the deployed OpenAPI spec: kubectl -n kubeflow port-forward svc/ml-pipeline 8888:8888 then curl localhost:8888/apis/v2beta1/openapi.json | jq '.paths."/apis/v2beta1/runs".get.parameters[] | select(.name=="page_size")' — look for maximum or x-kubernetes-max-items.
2. Empirical test: curl -G "$ENDPOINT/apis/v2beta1/runs" --data-urlencode "page_size=5000" --data-urlencode "sort_by=created_at desc" -H "Authorization: Bearer $(kubectl -n kubeflow get secret ml-pipeline-token -o jsonpath='{.data.token}' | base64 -d)" — observe whether the response caps total_size, returns an error, or honors the request.
3. Ordering sanity check: run two consecutive ListRuns calls with the same sort_by and the first call's next_page_token; verify that run IDs are strictly monotonic per the sort order and that no ID appears twice.

One Diagnostic Detail Needed

Which KFP backend version (or Kubeflow distribution + version) is deployed? The token encoding and default limits changed between v1 (Argo-based) and v2 (KFP-native) backends, and some managed services (Vertex AI Pipelines, AWS SageMaker Pipelines) impose their own caps.