Diagnosing Unexpected Deny Decisions in Ory Keto

Symptom: an allowed request returns deny

Your application calls Ory Keto to authorize an action, and Keto responds with a deny. The subject should have permission based on your access model, yet the decision is negative. The denial may appear as an HTTP 403, a JSON field such as allowed: false, or an audit entry without a clear reason.

Start here: reproduce the request directly against Keto's check endpoint and inspect Keto's server logs. Most unexpected denials come from a missing relation tuple, a malformed request, stale cached data, or Keto losing contact with its relation tuple store. A relation tuple is Keto's basic permission record: it states that a subject has a relation to an object within a namespace.

Cause and diagnostic table

Likely cause	What to look for	First check
Missing relation tuple	Subject lacks the expected permission in the store	Query Keto's read API for the tuple
Malformed request attributes	Request body missing required fields	Keto logs show a validation error
Stale cache or replica lag	Recent grant not reflected in decisions	Compare direct store read with Keto response
Policy store unreachable	Keto cannot contact the configured store	Keto logs show connection refused or timeout
Insufficient store credentials	Keto lacks read access to tuples	Logs show unauthorized or forbidden from the store
Client and server version mismatch	SDK sends fields the server ignores	Compare SDK and Keto versions
Clock skew	Time-based conditions fail	NTP status on Keto nodes and clients

Ordered checks

1. Reproduce with Keto's check endpoint

Run this from any host that can reach Keto. It is read-only and changes nothing.

curl -X POST 'http://${KETO_HOST}:4456/relation-tuples/check' \
  -H 'Content-Type: application/json' \
  -d '{
    "namespace": "${NAMESPACE}",
    "object": "${OBJECT}",
    "relation": "${RELATION}",
    "subject": { "id": "${SUBJECT}" }
  }'

• Where: terminal with network access to the Keto service
• Permissions: none beyond network reach; no cluster admin required
• Placeholders: KETO_HOST, NAMESPACE, OBJECT, RELATION, SUBJECT. The endpoint path can vary by Keto version; check your API reference.
• Expected: HTTP 200 with a JSON body containing an allowed field. A non-200 response or allowed: false confirms the denial.
• Risk: none; this is a read operation

2. Inspect Keto server logs

kubectl logs deployment/keto -c keto --tail=100

• Where: terminal with a valid kubectl context
• Permissions: RBAC to get pods and get logs
• Expected: lines mentioning deny, not found, connection refused, unauthorized, forbidden, or timeout
• Risk: none

3. Confirm the relation tuple exists

curl -X GET 'http://${KETO_HOST}:4456/relation-tuples?namespace=${NAMESPACE}&object=${OBJECT}&relation=${RELATION}&subject_id=${SUBJECT}'

• Where: any host with network reach to Keto
• Expected: a JSON array with at least one tuple. An empty array means the tuple is missing, which explains the deny.
• Risk: none

4. Check policy store connectivity and credentials

Keto stores relation tuples in a backend such as PostgreSQL, CockroachDB, or another configured store, reached through a data source name (DSN): a connection string that includes host, port, database, and authentication details.

• Look in Keto logs for store connection errors.
• Verify the DSN environment variable or config file matches the store endpoint used by other Keto instances.
• Confirm Kubernetes secrets or Vault tokens used by Keto have read scope for the tuple store.
• In multi-cluster deployments, all instances should share a consistent store URL.

5. Compare client and server versions

Check the Keto server version through its version or health endpoint, and check the client SDK version in your application's manifest or package manager. A mismatch can cause the SDK to send fields the server ignores, producing an unexpected deny.

6. Check clock synchronization

timedatectl

or

chronyc tracking

• Where: each Linux node running Keto and each client host
• Permissions: typically requires sudo or root
• Expected: synchronized status with a small clock offset
• Risk: none; read-only

Fixes tied to findings

• Missing tuple: write the missing tuple through Keto's write API, then re-run the check endpoint command above.
• Malformed request: correct the client payload so attribute names match the claims issued by your OIDC provider. Avoid hard-coding attribute names.
• Stale cache: restart the Keto pod or wait for the cache TTL to expire. Do not edit policy files in a running environment without a reload or restart; stale entries may linger.
• Store unreachable: fix network policy, DNS, or TLS configuration. Verify the store URL is consistent across clusters.
• Wrong credentials: rotate or update the Kubernetes secret or Vault token so Keto can read the tuple store.
• Version mismatch: align the client SDK with the Keto server version and review the changelog for breaking changes.
• Clock skew: enable or repair NTP or chrony on affected nodes, then restart Keto after the clock is synchronized.

Rollback when a fix changes state

Writing a tuple changes state. If the new grant is too broad, remove it with the inverse delete operation:

curl -X DELETE 'http://${KETO_HOST}:4456/relation-tuples?namespace=${NAMESPACE}&object=${OBJECT}&relation=${RELATION}&subject_id=${SUBJECT}'

Only run this after confirming the tuple caused the incorrect allow decision.

Escalation criteria

• The deny persists after all six checks and their corresponding fixes.
• Keto logs show internal server errors, panics, or repeated store failures.
• The issue affects multiple namespaces or clusters at the same time.
• You suspect corruption in the relation tuple store.
• If you use Ory Network (the managed offering), open a support ticket rather than modifying store credentials directly.

Limitations and how to verify the result

This guide covers common read-path denials. It does not cover the full Ory Keto authorization model, write-path conflicts, or Ory Network-specific behavior.

To verify a fix, re-run the check endpoint command and confirm allowed: true. Then monitor application traffic for 15 to 30 minutes to ensure denials stop recurring. If the problem returns, check for cache replication lag or a job that is rewriting tuples.