Support Boundaries for Kubernetes Service Account Token Renewal in Spark 3.5

Apache Spark's Kubernetes scheduler backend supports least-privilege authentication via Kubernetes service account tokens, yet the documented behavior for long-running applications that outlive these credentials remains ambiguous. In Spark 3.5, the default scheduler backend does not automatically refresh service account tokens or OAuth tokens for driver and executor pods; renewal is instead delegated to external sidecars or manual pod restarts. This leaves an unresolved boundary between Spark's internal authentication handling and cluster-level credential management.

For batch or Structured Streaming jobs that exceed the token lifetime, operators must choose between manual intervention, external renewal mechanisms, or restricting job duration. The core design question is whether Spark should integrate automatic token refresh into its Kubernetes scheduler or continue treating credential rotation as an external operational concern.

What is the documented support boundary for automatic service account token renewal in Spark's Kubernetes scheduler backend? Should long-running Spark applications on Kubernetes rely on external sidecars for token refresh, or is native integration expected? How does Spark 3.5 handle expired service account tokens for driver and executor pods without manual restarts?