Support Boundaries for Kubernetes Service Account Token Renewal in Spark 3.5

Question

Apache Spark's Kubernetes scheduler backend supports least-privilege authentication via Kubernetes service account tokens, yet the documented behavior for long-running applications that outlive these credentials remains ambiguous. In Spark 3.5, the default scheduler backend does not automatically refresh service account tokens or OAuth tokens for driver and executor pods; renewal is instead delegated to external sidecars or manual pod restarts. This leaves an unresolved boundary between Spark's internal authentication handling and cluster-level credential management.

For batch or Structured Streaming jobs that exceed the token lifetime, operators must choose between manual intervention, external renewal mechanisms, or restricting job duration. The core design question is whether Spark should integrate automatic token refresh into its Kubernetes scheduler or continue treating credential rotation as an external operational concern.

What is the documented support boundary for automatic service account token renewal in Spark's Kubernetes scheduler backend? Should long-running Spark applications on Kubernetes rely on external sidecars for token refresh, or is native integration expected? How does Spark 3.5 handle expired service account tokens for driver and executor pods without manual restarts?

Tasadduq Burney · Answer

In Apache Spark 3.5, the Kubernetes scheduler backend issues a service‑account token to driver and executor pods at launch and does not automatically refresh it for the duration of the job. When a token’s validity period expires—due to a short Kubernetes lease or cluster‑level rotation—the driver and executor processes lose authentication, causing the job to fail until the pod is restarted. The documented support boundary therefore does not include native token renewal; Spark treats credential rotation as an external operational concern.
Confirmed facts

Spark 3.5’s Kubernetes scheduler does not poll or refresh service‑account tokens automatically.
  Tokens are bound to the pod’s initial lifecycle and expire according to the service‑account token lifetime set by the cluster.
  Expired tokens result in authentication errors for Spark’s internal Kubernetes API calls from driver and executor components.
  No configuration flag or API in Spark 3.5 enables automatic token refresh or polling for updated tokens.

Practical steps for long‑running jobs

Deploy a sidecar container that monitors the service‑account token file (/var/run/secrets/kubernetes.io/serviceaccount/token) and triggers a pod restart or token rotation when the token changes.
  Alternatively, configure the Kubernetes service account with a long enough token lease to cover the expected job duration, and plan pod restarts between batches.
  For Structured Streaming jobs, use the spark.kubernetes.authenticate.driver.serviceAccountName setting to specify the account, then rely on cluster‑level mechanisms (e.g., CSI drivers, kube‑token‑refresh) to maintain validity.

Note: The behavior described reflects Spark 3.5 as released through mid‑2024; subsequent releases may introduce native token‑refresh capabilities. Operators should verify the specific Spark and Kubernetes versions in their environment.

Support Boundaries for Kubernetes Service Account Token Renewal in Spark 3.5

1 answer

Your perspective belongs here.

1 question comment

Token rotation vs. Spark awareness