Failed to obtain/user credentials when Kerberos ticket expires in Hadoop YARN

Question

When a Kerberos ticket used by a long-running Hadoop job expires, the client throws an AuthenticationFailedException with the message “Failed to obtain/user credentials”, causing the operation to fail unless the ticket is renewed manually or the job is restarted.
The goal is to allow such jobs to continue running without manual re‑login by enabling automatic ticket renewal through JAAS configuration or delegation‑token mechanisms, while keeping the cluster’s least‑privilege security posture intact.
However, the default Hadoop configuration does not renew tickets for user‑submitted jobs, and it is unclear whether setting renewTGT=true in the login module is sufficient across HDFS, YARN and MapReduce, or if additional delegation‑token renewal must be enabled. There is also uncertainty about whether any component silently falls back to simple authentication when tickets expire, which could weaken security.
What configuration is required to guarantee automatic Kerberos ticket renewal for user‑submitted MapReduce jobs? Does enabling renewTGT=true in the JAAS login module cover all Hadoop services, or must delegation‑token renewal be explicitly configured? Are there any version‑specific behaviors or side effects when relying on automatic renewal versus manual keytab refresh?

Tasadduq Burney · Answer

Answer first
For Hadoop 2.8+ and 3.x, automatic renewal of credentials for a user-submitted MapReduce v2 job requires a renewable Kerberos TGT and renewTGT=true in the JAAS login used by both the client and the ApplicationMaster. When those conditions hold, Hadoop’s KerberosRenewer automatically renews delegation tokens for HDFS and YARN. No separate delegation-token renewal flag is needed for those services. renewTGT=true alone does not help a non-renewable ticket and does not cover services that do not issue Hadoop delegation tokens.
Confirmed facts

KerberosRenewer is started by the client and by the ApplicationMaster when a job is submitted with Kerberos authentication. It renews delegation tokens obtained from NameNode and ResourceManager as long as the underlying TGT is renewable.
renewTGT=true in the JAAS login module allows the login module to renew the TGT before expiry. This is a prerequisite for the renewer to keep delegation tokens alive.
Hadoop 2.6+ introduced more reliable delegation token renewal for user jobs; prior releases required manual handling of mapreduce.job.credentials.binary and did not reliably renew tokens after the initial login.
Delegation token renewal is performed only for services that issue Hadoop delegation tokens: HDFS, YARN, Hive, HBase. Other frameworks must wire into Hadoop’s token renewal themselves.

Likely explanation for your error
"Failed to obtain/user credentials" with AuthenticationFailedException on a long job is most commonly a TGT expiry with no renewal path. If the ticket is non-renewable, or renewTGT is not set for the ApplicationMaster container, the renewer cannot refresh the TGT and delegation tokens expire. The job then fails on the next HDFS/YARN RPC.
Minimum steps for this case

Verify the principal is renewable: klist -f must show the R flag. A non-renewable ticket makes automatic renewal impossible.
Use a JAAS config with renewTGT=true for the client login and ensure the same settings are visible to the AM container. Example client JAAS:
com.sun.security.auth.module.Krb5LoginModule required
  useKeyTab=true
  keyTab="/path/to/user.keytab"
  storeKey=true
  useTicketCache=false
  renewTGT=true
  principal="user@REALM";

Do not disable renewal: ensure dfs.web.authentication.kerberos.renewal.disabled and similar flags are not set to true.
Verify renewal in logs: look for "KerberosRenewer: Renewing delegation token" and "TGT renewal scheduled" in the client and ApplicationMaster logs.

Version and side effects
With a renewable ticket, renewTGT=true avoids manual kinit and keytab relogin. With a non-renewable ticket, renewTGT=true causes repeated KerberosException retries and log noise. Relying on automatic renewal can hide keytab permission or deletion problems; if renewal fails, the job fails rather than silently falling back to simple authentication in a properly secured cluster. Manual keytab refresh is only required when the principal cannot be issued a renewable ticket.
One diagnostic detail that changes the recommendation: what Hadoop major/minor version is running and is the user’s Kerberos ticket issued with the renewable flag and a keytab available to the login context? Without a renewable ticket, no JAAS change will prevent expiry.

Failed to obtain/user credentials when Kerberos ticket expires in Hadoop YARN

1 answer

Answer first

Confirmed facts

Likely explanation for your error

Minimum steps for this case

Version and side effects

Your perspective belongs here.

0 question comments