Home / DE Professional practice test / Data Transformation, Cleansing, and Quality / Question

Databricks Data Engineer Professional · free practice question

A classic job uses Auto Loader to ingest JSON files where new fields can appear without notice and some fields occasionally arrive with type mismatches. The…

Exam-style question on Data Transformation, Cleansing, and Quality from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Data Transformation, Cleansing, and Quality Medium

A classic job uses Auto Loader to ingest JSON files where new fields can appear without notice and some fields occasionally arrive with type mismatches. The pipeline must keep running (no manual restarts), and unexpected fields must not be silently dropped. Which configuration best satisfies these requirements while minimizing custom state handling?

  1. ASet cloudFiles.schemaEvolutionMode='rescue' with a persistent cloudFiles.schemaLocation and checkpoint; new fields and type mismatches are kept in _rescued_data
  2. BSet badRecordsPath with a persistent cloudFiles.schemaLocation and checkpoint; new fields and type mismatches are kept in the bad records files for replay
  3. CSet cloudFiles.schemaEvolutionMode='addNewColumns' and remove the checkpoint; new fields are added to the target and type mismatches are cast to strings
  4. DSet an explicit schema with cloudFiles.schemaEvolutionMode='addNewColumns'; new fields are appended to that schema and type mismatches are written as nulls
Answer & explanation

Correct answer: A

WHY A: In rescue mode, Auto Loader does not fail the stream due to new columns and does not ignore unexpected fields; instead, it records unexpected columns and type/case mismatches in the rescued data column (default _rescued_data), allowing the pipeline to continue and letting you quarantine or inspect rescued content later. A persistent cloudFiles.schemaLocation and a stable checkpoint are required for reliable incremental processing and schema tracking, so no custom state handling is needed.

WHY NOT B: badRecordsPath stores malformed or corrupt records, but type mismatches and unexpected columns are handled by the rescued data column when it is configured; they are not necessarily treated as bad records, so this option confuses the corrupt-record sink with schema drift handling.

WHY NOT C: Removing the checkpoint breaks exactly-once guarantees and can cause reprocessing, and the checkpoint is what lets Auto Loader track state without custom code. addNewColumns also stops the stream when a new column arrives (it updates the schema and the stream typically needs a restart), and capturing mismatched values is the job of the rescued data column, not of a cast to strings.

WHY NOT D: When you provide an explicit schema, addNewColumns is not allowed in the same way (and defaults differ); you typically need schema hints or other handling. With an explicit schema and no rescued data column, mismatched values end up as nulls, which silently loses data the requirements say must be kept; this is not the keep-running configuration.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →