Home / DE Professional practice test / Developing Code for Data Processing using Python and SQL / Question

Databricks Data Engineer Professional · free practice question

A data engineering team is operating a production Lakeflow Spark Declarative Pipeline in `TRIGGERED` mode that processes financial transaction data from Auto…

Exam-style question on Developing Code for Data Processing using Python and SQL from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Developing Code for Data Processing using Python and SQL Medium

A data engineering team is operating a production Lakeflow Spark Declarative Pipeline in TRIGGERED mode that processes financial transaction data from Auto Loader. After a routine deployment, the pipeline fails with the following error in the Event Log: SCHEMA_MISMATCH: Schema of data does not match schema of streaming table: expected 'amount DECIMAL(18,2)' but got 'amount DOUBLE'. The upstream source team has confirmed they changed the amount field type from DECIMAL(18,2) to DOUBLE in a recent data export. The pipeline must resume processing as quickly as possible without data loss. What is the CORRECT remediation sequence?

  1. AUpdate the streaming table definition to change the amount column type from DECIMAL(18,2) to DOUBLE to match the new source schema. Run a full refresh of the pipeline to rebuild the streaming table from scratch with the new schema, then resume normal TRIGGERED mode operation.
  2. BIn the pipeline settings, change cloudFiles.schemaEvolutionMode from none to addNewColumns and restart the pipeline. The addNewColumns mode handles type changes as well as new columns by widening the schema to accommodate both type representations of amount.
  3. CRun ALTER TABLE silver_transactions ALTER COLUMN amount TYPE DOUBLE on the pipeline's output Delta table from a Databricks SQL notebook, then restart the pipeline without a full refresh. The type change applied directly to the Delta table resolves the mismatch without reprocessing.
  4. DUpdate cloudFiles.schemaEvolutionMode to rescue in the pipeline configuration. Records with the new DOUBLE type for amount are then stored in the _rescued_data column as JSON strings instead of failing, and the pipeline resumes while a permanent schema fix is prepared.
  5. EDelete the pipeline's checkpoint directory in DBFS and restart the pipeline. Deleting the checkpoint forces Auto Loader to re-scan the files from the beginning and re-infer the schema from the latest data, and the streaming table is rebuilt with the DOUBLE type on that run.
Answer & explanation

Correct answer: A

WHY A: A SCHEMA_MISMATCH error where an existing column's data type has changed (not just a new column added) requires a schema-breaking change to the streaming table definition. The procedure is: (1) update the streaming table's definition to reflect the new type, (2) perform a Full Refresh (pipeline reset, from the UI or databricks pipelines start --full-refresh), which truncates and rebuilds the target table so the new schema is applied consistently from the beginning, then resume TRIGGERED operation. This is the Lakeflow SDP procedure for type-changing schema evolution.

WHY NOT B: addNewColumns handles the addition of new fields. It does not handle type changes to existing columns, so a DECIMAL-to-DOUBLE change is not resolved by it.

WHY NOT C: Altering the Delta table schema directly via ALTER TABLE outside the pipeline can cause the Lakeflow SDP engine to detect a schema conflict between the pipeline definition and the physical table schema, potentially causing further errors. The pipeline definition and the target table schema are updated in a coordinated way via a Full Refresh.

WHY NOT D: rescue mode captures records that don't match the current schema into _rescued_data. The amount field would land as a JSON string there rather than being processed correctly, and rescued records require manual reprocessing, so this is not a quick resume strategy.

WHY NOT E: Manually deleting the checkpoint directory corrupts the pipeline's incremental state. Databricks manages checkpoints as part of the pipeline metadata, and a Full Refresh through the pipeline UI/API is the supported way to reset pipeline state.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →