Home / DE Professional practice test / Data Ingestion & Acquisition / Question

Databricks Data Engineer Professional · free practice question

A streaming pipeline ingests events and uses `foreachBatch` to write the same micro-batch into two Delta tables (a raw table and an audit table). During…

Exam-style question on Data Ingestion & Acquisition from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Data Ingestion & Acquisition Medium

A streaming pipeline ingests events and uses foreachBatch to write the same micro-batch into two Delta tables (a raw table and an audit table). During intermittent failures, the job retries a batch and occasionally writes duplicates. Which configuration best makes the batch writes idempotent so retries do not duplicate data?

  1. ASet the output mode to append and remove the checkpoint location, letting Delta Lake deduplicate identical rows on each retried write to both tables.
  2. BUse COPY INTO inside foreachBatch with COPY_OPTIONS('force'='true'), letting Delta reload the same files safely whenever the batch is retried.
  3. COn each write inside foreachBatch, set option('txnAppId', <stable_app_id>) and option('txnVersion', batch_id), letting Delta skip repeated writes.
  4. DEnable option('mergeSchema', 'true') on both writes inside foreachBatch, letting schema evolution reconcile the rows written when a batch is retried.
Answer & explanation

Correct answer: C

WHY C: Delta supports idempotent writes using the (txnAppId, txnVersion) pair. When a batch is retried, reusing the same app ID and batch ID lets Delta identify duplicate writes for that pair and skip them. This is especially important when doing multiple writes inside foreachBatch.

WHY NOT A: Checkpoints are required for Structured Streaming correctness; Delta does not deduplicate rows by itself, and removing checkpoints makes reprocessing more likely.

WHY NOT B: COPY INTO is for loading from file locations; force=true disables its idempotency and can introduce duplicates.

WHY NOT D: Schema evolution settings do not address at-least-once retry behavior; duplicates are a write idempotency problem, not a schema problem.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →