Diagnosing Rancher v2.7+ OIDC Authentication Failures

Problem and Takeaway

Rancher's OIDC integration breaks in predictable ways: redirect URI mismatches, missing claims, redirect loops, JWKS fetch errors, group sync gaps, and session loss in HA. The root cause is almost always a configuration drift between Rancher, the identity provider, and the network path between them. This guide gives you a symptom-to-cause map, ordered verification steps, targeted fixes, and clear escalation thresholds so you can resolve the issue without guessing.

Recognizable Symptoms

Cause and Diagnosis Matrix

Ordered Diagnostic Checks

1. Verify server-url matches IdP redirect URI exactly. In Rancher UI: Settings → server-url. Compare scheme, host, port, and path (/v3-public/oidc/callback) with the IdP's registered redirect URI. A trailing slash or port difference breaks the flow.
2. Capture OIDC callback payload. Enable debug logging temporarily: set CATTLE_DEBUG=true on the Rancher deployment, trigger a login, then check Rancher pod logs for oidc callback JSON. Risk: full tokens appear in logs — disable immediately and redact before sharing.
3. Test JWKS endpoint reachability. From a Rancher pod in the cattle-system namespace:
   

   kubectl exec -it <rancher-pod> -n cattle-system -- curl -s <issuer-url>/.well-known/jwks.json | jq .

   Expect HTTP 200 and a valid JWKS JSON. Failure indicates network policy, firewall, or DNS block.
4. Validate claim mappings against actual IdP token. Decode the ID token (e.g., via jwt.io or jq -R 'split(".") | .[1] | @base64d | fromjson' <<< "$TOKEN"). Compare claim names to Rancher OIDC config (UI → Authentication → OIDC → User Claim / Group Claim fields).
5. Confirm proxy headers and TLS termination config. Ensure the proxy forwards X-Forwarded-Proto: https and X-Forwarded-Host, preserves the Host header, and Rancher's server-url uses the external HTTPS URL. Set CATTLE_TLS_MODE=external on the Rancher deployment.
6. Verify Redis session store in HA setups. After a successful login, check Redis for session keys:
   

   kubectl exec -it <redis-pod> -- redis-cli KEYS 'oidc:*'

   Keys should exist. Delete the Rancher pod that handled the login and confirm the session survives on another replica.

Targeted Fixes

Redirect URI Mismatch

Align Rancher server-url and IdP redirect URI exactly. If you change server-url, restart the Rancher pod (rolling restart of the deployment). Schedule during a maintenance window — existing user sessions are invalidated.

Missing Claims

Adjust authConfig.oidc.userNameClaim and groupClaim to match the IdP's token. Common IdP quirks:
- Keycloak: add a Group Membership mapper to emit groups claim.
- Azure AD: request groups claim in the app manifest.
- Okta: include groups scope in the authorization request.
After changes, test with a fresh login and verify debug callback payload.

Redirect Loop Behind Proxy

1. Set CATTLE_TLS_MODE=external on all Rancher replicas.
2. Configure proxy to forward X-Forwarded-Proto, X-Forwarded-Host, and preserve Host header.
3. Ensure Rancher server-url is the external HTTPS URL (e.g., https://rancher.example.com).
4. Restart Rancher pods.

JWKS Fetch Errors

1. Allow egress from Rancher pods to the IdP domain in network policies / firewall rules.
2. If keys rotated, restart Rancher pods to refresh the JWKS cache (Rancher caches JWKS until pod restart).
3. For recurring rotation issues, consider a sidecar that periodically pre-fetches JWKS or a shorter cache TTL if Rancher adds one in future versions.

Group Sync Gaps

If the IdP cannot emit nested groups or token size is exceeded, enable IdP-side group filtering (custom claims) to send only relevant groups. For complex hierarchies, switch to Rancher's LDAP/AD group sync instead of relying on OIDC group claims.

HA Session Loss

1. Deploy a Redis instance (standalone or clustered) accessible from all Rancher replicas.
2. Set the following env vars on every Rancher replica:
   CATTLE_OIDC_SESSION_REDIS_HOST, CATTLE_OIDC_SESSION_REDIS_PORT, CATTLE_OIDC_SESSION_REDIS_DB, and optionally CATTLE_OIDC_SESSION_REDIS_PASSWORD.
3. Verify with the Redis KEYS 'oidc:*' check after login and pod deletion test.

Escalation Criteria

Open a Rancher support ticket when:

• IdP token structure cannot be changed and claim mapping fails after two documented iterations.
• JWKS rotation causes recurring outages despite pod restarts and verified network egress.
• HA session loss persists with a verified Redis configuration (keys present, replicas connected).
• You suspect a Rancher OIDC code defect — provide debug logs (redacted), Rancher version (kubectl get deployment rancher -n cattle-system -o jsonpath='{.spec.template.spec.containers[0].image}'), IdP type and version.

Verification Checklist

Limitations and Cautions

• Changing server-url requires a Rancher pod restart and invalidates all existing user sessions. Plan for a maintenance window.
• Debug logging (CATTLE_DEBUG=true) exposes full OIDC tokens in logs. Disable immediately after capture; redact tokens before sharing any logs.
• Network policies blocking egress to the IdP JWKS endpoint are a frequent silent failure. Always test with curl from the Rancher pod.
• Configuration source: Rancher v2.7+ uses the dynamic schema authConfig.oidc. Legacy env vars (CATTLE_OIDC_*) are still read but deprecated. Avoid mixing both — pick one method.
• PKCE unsupported: Rancher's OIDC implementation does not support PKCE. Public clients (SPAs) cannot use Rancher OIDC directly; use a backend-for-frontend pattern.