Home / GenAI Engineer practice test / Data Preparation / Question

Databricks Generative AI Engineer Associate · free practice question

An engineering group intends to establish a Databricks pipeline for incrementally processing text chunks. Daily additions of PDF documents occur within a Unity…

Exam-style question on Data Preparation from the free GenAI Engineer practice test, with the answer and a full explanation below.

Updated November 11, 2026

Data Preparation

An engineering group intends to establish a Databricks pipeline for incrementally processing text chunks. Daily additions of PDF documents occur within a Unity Catalog Volume. This pipeline is required to: (1) identify solely the newly appeared PDFs in each execution cycle, (2) extract and segment their textual content, (3) add new chunk entries to an established catalog.rag.chunks_bronze Delta table without disturbing existing records, and (4) ensure no duplicate chunks are created if a document is inadvertently processed more than once. What sequence of operations would accurately satisfy these four conditions?

  1. AIngest with Auto Loader (cloudFiles) so its checkpoint picks up new files, append the chunks to chunks_bronze, and afterwards run MERGE INTO on (doc_id, chunk_index).
  2. BTrack handled paths in a processed_files Delta table, find new PDFs with a LEFT ANTI JOIN against the Volume listing, and MERGE INTO chunks_bronze on (doc_id, chunk_index).
  3. CRead every file in the Volume on each run, re-chunk the full set, and overwrite chunks_bronze, relying on Delta time travel to restore earlier versions if needed.
  4. DAppend the new chunks with saveAsTable and no deduplication step, relying on Delta Lake ACID transaction isolation to reject rows written by a repeated job.
  5. EList the Volume with dbutils.fs.ls on each run, track handled files in a Python text file, and append the new chunks, relying on Delta Lake to drop identical rows.
Answer & explanation

Correct answer: B

WHY B is correct: A processed_files tracking table with a LEFT ANTI JOIN identifies only the PDFs not yet handled, and writing the chunks with MERGE INTO on the (doc_id, chunk_index) key adds new rows without touching existing ones while updating matches instead of inserting duplicates when a document is reprocessed. Together these steps cover all four requirements.

WHY NOT E: Delta Lake has no feature that detects and drops identical rows on append, so reprocessing a document would insert its chunks a second time. A side text file is also a fragile way to track which files were handled.

WHY NOT A: Auto Loader's checkpoint does handle incremental file discovery, but this sequence appends the chunks to the target table before merging. If a document is processed again, the append has already inserted the duplicate rows, and the later MERGE does not remove them, so the fourth requirement is not met.

WHY NOT C: Overwriting the table on every run violates the requirement to leave existing records untouched and reprocesses the whole corpus each time, which is not incremental. Time travel allows recovery but does not make an overwrite an incremental load.

WHY NOT D: ACID guarantees make each write atomic and isolate concurrent transactions, but they do not stop a later run from appending the same rows again. Plain append mode has no built-in deduplication.

Take the full GenAI Engineer practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Generative AI Engineer Associate practice test. Take it and get a section-by-section score.

Start the free test →