A Delta table fed by many small streaming micro-batches has accumulated thousands of tiny files, and interactive reads over it have grown slow. Metrics show time is spent opening files rather than on skew or memory spill. What best diagnoses and remediates this?
Answer & explanation
Correct answer: C
WHY C: Thousands of tiny files from frequent micro-batches is the small-file problem; OPTIMIZE bin-packs them into larger files, and auto-compaction plus optimizeWrite prevent new small files. WHY NOT A: shuffle.partitions governs shuffle width, not output file sizes on a streaming Delta sink, so it does not fix the tiny files. WHY NOT B: VACUUM only removes old unreferenced files; it does not compact or merge the current small files. WHY NOT D: caching hides the cost for cached reads but leaves the small-file layout and its open overhead unfixed. WHY NOT E: repartitioning the write to more partitions produces even more small files, worsening the problem.