#executor-memory

Executor memory overhead change Spark 3.x local mode to YARN Kubernetes

Executor memory limits that are sufficient on a developer workstation do not map consistently to container limits in production. In Spark 3.x, spark.executor.memory defines heap while spark.executor.memoryOverhead adds a non-heap allowance, and the sum is enforced by YARN and Kubernetes cgroups. Local[*] mode runs driver and executors in one JVM and does not

Community & Careers · Editorial question

yarnkubernetesexecutor-memory