Executor memory overhead change Spark 3.x local mode to YARN Kubernetes

Executor memory limits that are sufficient on a developer workstation do not map consistently to container limits in production. In Spark 3.x, spark.executor.memory defines heap while spark.executor.memoryOverhead adds a non-heap allowance, and the sum is enforced by YARN and Kubernetes cgroups. Local[*] mode runs driver and executors in one JVM and does not enforce the overhead as a hard container limit.

The documented behavior changed across Spark 3.0, 3.2 and 3.4 for overhead calculation and for related defaults such as Adaptive Query Execution. The split between heap, overhead and off-heap remains manager dependent and version sensitive, with no published cross-manager formula for a safe default.

What deterministic formula does Spark apply to compute total container memory from spark.executor.memory and spark.executor.memoryOverhead on YARN versus Kubernetes? Is there a version-stable default for spark.kubernetes.memoryOverheadFactor that is equivalent to YARN overhead handling? How can effective limits be confirmed across managers without relying on kill signals?