Question
401 Unauthorized on Kubeflow dashboard after OIDC token revocation
Mira Byte
0 reputation · 16 Jun 2023, 23:28 UTC
90.6K views1
Loading…
0 reputation · 16 Jun 2023, 23:28 UTC
In a Kubeflow environment utilizing Istio for authentication, the auth-proxy validates OIDC tokens at the request level. While Kubernetes RBAC manages permissions based on OIDC groups, there is a discrepancy between the identity provider's state and the active Istio session.
When a user is deactivated or their credentials are rotated in the OIDC provider, the Kubeflow UI often remains accessible due to local session state. However, subsequent API calls to the dashboard return a 401 Unauthorized error. Because the 'auth' component lacks a native mechanism to force a global invalidation of specific user tokens, these sessions remain active until the JWT naturally reaches its TTL.
Is there a documented method to trigger immediate revocation of active Istio sessions for specific users without implementing custom Envoy filters? Additionally, how can the system be configured to verify the current status of a token against the provider rather than relying solely on JWT expiration dates?
35275 reputation · 17 Jun 2023, 09:32 UTC
kubeflow401 Unauthorized error occurs when the dashboard UI attempts an API call using the local session, but the backend service (or a refreshed RBAC check) identifies the identity as no longer valid or the proxy rejects the request.
Since there is no documented method to revoke sessions without Envoy filters, you can use the following approaches to force invalidation:
dex-session or similar) in the browser. This forces the client to re-authenticate against Dex, which will then check with the IdP.token-lifetime in the Dex configuration. This limits the "window of opportunity" where a revoked token remains active, though it increases the load on your IdP.To move beyond relying solely on JWT expiration dates, you must configure the system to perform real-time validation. This is typically achieved through:
Diagnostic Question: To provide a more specific configuration, is the 401 error returned by the Istio-ingress gateway or by a specific Kubeflow microservice (like the notebook or dashboard API)?
Use comments to ask for clarification. Post a solution as an answer.
35,275 reputation · 17 Jun 2023, 04:41 UTC
To address the second part of the query regarding real-time status verification, it is important to distinguish between stateless JWT validation and token introspection.
By default, Istio's RequestAuthentication validates JWTs locally using the provider's public keys (JWKS). This is high-performance but ignores revocation until the token expires. To verify the current status against the provider, the architecture must be shifted to use an OIDC Introspection endpoint (RFC 7662). This requires the auth-proxy to make a synchronous API call to the IdP for every request or at a defined interval.
To diagnose if a session is relying on a stale cached token versus a revoked one, you can:
Authorization header in the browser's Network tab to see if the Bearer token is still being sent.oauth2-proxy or auth-proxy logs for specific 401 responses originating from the IdP's introspection endpoint rather than local cryptographic failure.