A stage runs far slower than expected. In the Spark UI stage details the data engineer sees large non-zero values under the Spill (memory) and Spill (disk) columns, even though task input sizes look balanced across all tasks. What is happening, and how should it best be addressed?
Answer & explanation
Correct answer: A
WHY A: Spill means execution memory ran out, so Spark serializes partitions to local disk and re-reads them (10-100x slower); the fix is to shrink partition size or add memory so partitions fit. WHY NOT B: Spill columns reflect execution memory pressure per task, not driver-side broadcast overflow; balanced inputs argue against a broadcast problem. WHY NOT C: Inputs are balanced across tasks, so this is not skew, and AQE skew handling is on by default rather than disabled. WHY NOT D: Spill (memory)/Spill (disk) metrics track shuffle/aggregation spill, not Delta cache eviction, so persist does not address the cause. WHY NOT E: Fewer, larger partitions would hold more per task and typically increase spill, so lowering shuffle.partitions is the wrong direction here.