Does VTTablet credential caching delay password expiration enforcement?

Answer to the Question

In Vitess 2.x, VTTablet does not immediately enforce a MySQL password expiration that occurs while a connection is still held in its internal connection pool. The cached connection continues to be used until the pool entry is evicted or the tablet is restarted. Password changes or expirations are only checked on the next authentication attempt that forces VTTablet to refresh its cached credentials.

Likely Explanation

VTTablet stores user credentials in a local cache that is refreshed only when a new authentication request is made or when the cache entry expires. Because the pool holds open connections, those connections are not re‑authenticated against MySQL until they are closed or the tablet restarts. Therefore, a password that has expired on the MySQL server can remain usable via an existing pool connection for the duration of the cache lifetime.

Confirmed Facts (as of Vitess 2.x)

• VTTablet forwards authentication to MySQL; MySQL returns error 1820 when the password has expired.
• The connection pool does not automatically invalidate an entry when the underlying MySQL account changes.
• VTTablet’s credential cache timeout is configurable (default is typically 5–15 minutes in many deployments), but the exact value depends on the tablet’s configuration.
• When the cache expires or the tablet is restarted, the next authentication will be performed against MySQL and will immediately enforce the new password policy.

Steps to Verify and Mitigate

1. Check Cache Timeout: Open vtctld.cfg or the tablet’s admin console and locate the credential_cache_timeout setting. Document its current value.
2. Simulate a Password Expiration: In a test environment, set a short password expiration on the MySQL user, then change the password or force it to expire.
3. Test Existing Connections: From a client that has already authenticated through VTTablet, attempt a query immediately after the password change. If the query succeeds, the cached connection is still in use.
4. Force Cache Eviction: Either wait for the cache timeout to elapse or manually clear the cache (e.g., by restarting the tablet or using a reset command if available). Then attempt a new query; authentication should now fail with error 1820 until the new password is supplied.
5. Adjust Configuration: If immediate enforcement is required, reduce the credential_cache_timeout to a very short interval (e.g., 60 seconds) or disable caching entirely if the deployment permits.

What If the Cache Timeout Is Not Set?

In some older VTTablet builds, the cache timeout might not be exposed in configuration files. In such cases, the default may be longer, leading to a larger window of delayed enforcement. Verify by checking the tablet’s logs for entries related to credential cache eviction.

Additional Diagnostic Detail Needed?

If you are unsure whether your tablet uses the credential_cache_timeout setting, please check the current value in the configuration file or admin console. Knowing this will help determine the exact propagation delay for credential updates.