Delegation token expired: ResourceManager rejects application master operations

Problem Context

In a secure YARN cluster, an application master (AM) relies on a delegation token issued by the ResourceManager (RM). The token is valid until its configured max-lifetime, after which the RM will reject any subsequent AM requests even if the client still holds a valid Kerberos keytab.

Goal

Design a reliable strategy for long‑running jobs so they do not fail when a delegation token reaches its maximum lifetime, while preserving the least‑privilege principle.

Constraints & Uncertainties

• The RM’s token secret manager enforces max-lifetime and renew-interval independently of the Kerberos ticket lifetime.
• When the token can no longer be renewed (due to hitting max-lifetime or the renewer’s Kerberos session expiring), YARN does not prescribe whether the client should fail closed or silently re‑login from a keytab and obtain a fresh token.
• Clock skew between RM and client can cause a token that the client considers current to be rejected.

Specific Questions

1. What is the intended policy for the RM when a delegation token reaches its maximum lifetime: should the AM fail immediately, or should it trigger a silent re‑authentication to obtain a new token?
2. How does YARN handle token renewal failures when the client’s Kerberos TGT has expired but the keytab remains available?
3. Is there a recommended configuration or client‑side logic to mitigate clock‑skew induced token rejections while maintaining least‑privilege delegation?