Home / DE Associate practice test / Troubleshooting, Monitoring, and Optimization / Question

Databricks Data Engineer Associate · free practice question

A Delta table fed by many small streaming micro-batches has accumulated thousands of tiny files, and interactive reads over it have grown slow. Metrics show…

Exam-style question on Troubleshooting, Monitoring, and Optimization from the free DE Associate practice test, with the answer and a full explanation below.

Updated November 10, 2026

Troubleshooting, Monitoring, and Optimization

A Delta table fed by many small streaming micro-batches has accumulated thousands of tiny files, and interactive reads over it have grown slow. Metrics show time is spent opening files rather than on skew or memory spill. What best diagnoses and remediates this?

  1. AThis is shuffle under-partitioning; raise spark.sql.shuffle.partitions so the writes spread across more tasks, which coalesces the many tiny files into fewer large ones and restores the fast interactive reads.
  2. BThe tiny files are stale, so run VACUUM on the table, which compacts the many small files into larger ones and reclaims their space, thereby eliminating the per-file open overhead slowing the interactive reads.
  3. CThis is the small-file problem; run OPTIMIZE to bin-pack the tiny files into larger ones, and enable auto-compaction and optimizeWrite so future micro-batches stop producing thousands of new small files at all.
  4. DThe reads are slow only from repeated recomputation, so cache the Delta table in cluster memory; subsequent reads then bypass the many small files entirely and the open overhead no longer affects the query time.
  5. EThe table has too few output files, so repartition the write to many more partitions, producing extra files per micro-batch so each read task handles a smaller file and the total scan then finishes much faster.
Answer & explanation

Correct answer: C

WHY C: Thousands of tiny files from frequent micro-batches is the small-file problem; OPTIMIZE bin-packs them into larger files, and auto-compaction plus optimizeWrite prevent new small files. WHY NOT A: shuffle.partitions governs shuffle width, not output file sizes on a streaming Delta sink, so it does not fix the tiny files. WHY NOT B: VACUUM only removes old unreferenced files; it does not compact or merge the current small files. WHY NOT D: caching hides the cost for cached reads but leaves the small-file layout and its open overhead unfixed. WHY NOT E: repartitioning the write to more partitions produces even more small files, worsening the problem.

Take the full DE Associate practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Data Engineer Associate practice test. Take it and get a section-by-section score.

Start the free test →