Diagnosing Vault AppRole 403 invalid credentials errors

Recognizable condition

When a client attempts to authenticate with Vault using the AppRole method, the request POST /v1/auth/approle/login returns HTTP 403 with the message "invalid credentials" even though the supplied role_id and secret_id were previously known to work.

Cause / diagnostic table

Possible cause	What to look for
secret_id expired or reached max_uses	Vault rejects the credential because its TTL elapsed or use counter is zero.
role_id deleted or role disabled	The role associated with the role_id no longer exists or has been set to disabled.
bound_cidr_list excludes client IP	The role restricts login to specific CIDR blocks and the client’s address is outside.
secret_id belongs to a different role	The secret_id was generated under another role name; lookup will show a mismatch.
Audit log missing the attempt	No login entry appears, indicating the request never reached the auth method (network or seal issue).
Clock skew between client and Vault	Vault rejects tokens if the request timestamp is outside the allowed skew window.

Ordered checks

1. Confirm the role exists and its constraints

   Run on a Vault admin host (requires a token with read capability on auth/approle/role/*):

   vault read auth/approle/role/<role-name>

   Check the output for disabled, bound_cidr_list, secret_id_ttl, and max_uses. If the role is missing or shows disabled = true, the role_id is invalid.

2. Capture the exact error from the client

   From the client machine, run the login call (replace placeholders):

   curl --request POST \
     --data '{"role_id":"<role-id>","secret_id":"<secret-id>"}' \
     <vault-addr>/v1/auth/approle/login

   Note the HTTP status and JSON error message. A 403 with "invalid credentials" confirms the symptom.

3. Inspect the secret_id metadata

   With an admin token (needs write on auth/approle/role/*/secret-id/lookup):

   vault write auth/approle/role/<role-name>/secret-id/lookup secret_id=<secret-id>

   The response includes ttl (remaining seconds) and uses_left. If uses_left = 0 or ttl ≤ 0, the secret_id is exhausted or expired.

4. Check the audit device for the login attempt

   If a file audit device is enabled (see verification section), grep for the attempt:

   grep approle /tmp/vault-audit.log | grep 403

   Absence of any line suggests the request never reached the auth layer (network, seal, or listener issue).

5. Compare system clocks

   On the client and each Vault server node, run:

   date -u

   Compute the offset; it should be under a few seconds. Large skew can cause Vault to reject the request timestamp.

6. Verify client IP against bound_cidr_list

   If the role shows a bound_cidr_list, compute whether the client’s outbound IP (as seen by Vault) falls within any of those CIDR blocks. You can obtain the client’s external IP via a service like curl https://ifconfig.me and compare manually or with ipcalc.

Fixes tied to findings

1. Rotate or refresh the secret_id

   If uses_left = 0 or TTL expired, generate a new secret_id for the role:

   vault write -f auth/approle/role/<role-name>/secret-id

   Distribute the new value to the client via a secure channel. Remember to treat the secret_id like a password: never log it, restrict its transmission to trusted parties, and rotate it according to your security policy.

2. Adjust secret_id lifetime or use limit

   To avoid frequent rotation, increase the role’s defaults:

   vault write auth/approle/role/<role-name> secret_id_ttl=24h secret_id_num_uses=0

   Setting secret_id_num_uses=0 disables the use limit; secret_id_ttl controls how long a generated secret_id remains valid. Test changes in a non‑production namespace first.

3. Restore or recreate a missing role

   If the role is disabled or absent, re‑enable it or recreate it with the same policies:

   vault write auth/approle/role/<role-name> disabled=false token_policies="<policy-list>"

   After re‑creation, generate a fresh secret_id and update the client.

4. Modify bound_cidr_list or adjust network routing

   If the client’s IP is outside the allowed CIDR, either add the client’s network to the list or route traffic through an approved proxy:

   vault write auth/approle/role/<role-name> bound_cidr_list="10.0.0.0/8,192.168.1.0/24"

   Validate the CIDR notation before applying; an incorrect block can lock out all legitimate clients.

5. Re‑register the secret_id under the correct role

   If the secret_id belongs to another role, delete it (if possible) and create a new one under the intended role, then update the client.

6. Synchronize time with NTP

   Ensure both client and Vault nodes are configured to use a reliable NTP source. Check the offset after synchronization:

   chronyc tracking   # or ntpq -p

   Keep the offset under 500 ms to stay within Vault’s default clock‑skew tolerance.

Escalation criteria

• The audit device shows no login attempt despite verified network reachability to the Vault address.
• Multiple roles or namespaces begin returning 403 for valid credentials simultaneously.
• Vault reports a sealed state or Raft layer unhealthy (vault status shows sealed = true or ha_enabled = true with no leader).
• Correct credentials fail cluster‑wide after a restart or reload of the auth method.
• Suspected platform issue (e.g., load balancer misconfiguration, firewall rule change) that affects traffic to the Vault listener.

When any of these conditions appear, involve the Vault operations team, check the underlying storage backend, and consider opening a support ticket with HashiCorp if using Enterprise.

Verification

1. Enable a file audit device (if not already present):

   vault audit enable file path=/tmp/vault-audit.log

2. Trigger a login attempt and confirm the failure is recorded:

   grep approle /tmp/vault-audit.log | grep 403

3. Read the role definition to verify the values you changed:

   vault read auth/approle/role/<role-name>

4. Use the secret-id lookup endpoint to ensure the new secret_id has the expected TTL and use count:

   vault write auth/approle/role/<role-name>/secret-id/lookup secret_id=<new-secret-id>

5. Check clock skew after NTP sync:

   date -u   # on client
   ssh vault-node-01 date -u   # repeat on each node

Limitations

The exact wording of the 403 error, default TTLs, and the availability of the secret-id/lookup endpoint can vary between Vault OSS and Enterprise releases. Always consult the release notes for your deployed version before altering role parameters. The audit device must be enabled and have sufficient permissions for the troubleshooting user; otherwise, the absence of log entries does not guarantee a network issue.