Executor memory in Spark governs how much heap space the executor JVM uses for task execution, data caching, and internal data structures. Understanding this memory helps you tune jobs, avoid out-of-memory errors, and use cluster resources efficiently.
Below is a concise overview of executor memory components, configuration levers, and practical tuning guidance.
| Component | Purpose | Configuration Parameter | Typical Tuning Guidance |
|---|---|---|---|
| Execution Memory | Used for shuffle, sort, and hash joins | spark.executor.memoryOverhead |
Reserve at least 10–15% of executor memory for overhead |
| Storage Memory | Caches RDDs, DataFrames, and broadcast blocks | spark.memory.storageFraction |
Default 0.5; adjust to favor caching vs execution |
| User Memory | Spark internal data structures and user code metadata | Derived from total executor memory | Largely managed automatically, but affects usable space |
| Reserved/Other | JVM overhead, native allocations, and safety margin | spark.executor.memoryOverhead |
Increase for large records, custom serializers, or native calls |
Execution Memory Management in Spark
Execution memory powers shuffle aggregation, joins, and sorting in Spark. The system dynamically shares this region between execution and storage using a unified memory model. If execution demand spikes, storage blocks can be evicted to make room, and vice versa, within the assigned region.
You control the total size via spark.executor.memory, while partitioning between execution and storage is managed by spark.memory.fraction and spark.memory.storageFraction. Raising the fraction increases execution headroom at the cost of less cache capacity per executor.
Executor Memory Overhead and Sizing
Memory overhead captures non-heap usage, including native memory for networking, buffers, and off-heap data structures. Insufficient overhead leads to container failures even when heap usage appears modest. Always set spark.executor.memoryOverhead explicitly when working with large records, custom serializers, or native libraries.
Start with the default, then add increments based on observed native usage and GC logs. For high-throughput workloads or structured streaming with large rows, allocate at least 1–2 GB per executor beyond the heap target.
Monitoring and Diagnosing Memory Issues
Use the Spark UI to track storage memory, execution memory utilization, and spill metrics. Spill-to-disk indicates execution memory pressure, while frequent GC or failed containers points to insufficient overhead or heap pressure.
Executor logs and external metrics from the JVM help correlate OOM events with specific stages. Adjust spark.executor.memory, spark.executor.memoryOverhead, and storage fractions iteratively, validating changes with representative workloads.
Best Practices and Key Takeaways
- Set
spark.executor.memoryOverheadexplicitly and size it for native usage. - Monitor spills and GC to detect execution or storage memory pressure.
- Adjust storage fraction when caching dominates workload performance.
- Scale executor count when cluster saturation is the bottleneck.
- Validate changes with realistic data sizes and stage-level metrics.
FAQ
Reader questions
How can I tell if my executor memory is undersized for my workload?
Watch for frequent spills to disk, long GC pauses, or repeated container failures in the cluster manager, which typically indicate that either heap or overhead memory is too low.
What is the right ratio between execution and storage memory?
Default storage fraction is 0.5, which reserves half of the unified region for caching. Increase storage fraction if your job is cache-bound; raise execution fraction for heavy shuffles or large joins.</p
Should I increase executor memory or number of executors to handle larger shuffles?
First tune spark.executor.memory and spark.executor.memoryOverhead to reduce spills and OOMs. If the cluster is saturated, scale out by adding more executors to parallelize work and avoid stragglers.
How does memoryOverhead interact with off-heap memory settings?
Memory overhead covers native buffers used by off-heap memory, networking, and JVM structures. If you enable off-heap storage or use native code, increase memoryOverhead to accommodate those allocations safely.