A data engineering team is operating a production Lakeflow Spark Declarative Pipeline in TRIGGERED mode that processes financial transaction data from Auto Loader. After a routine deployment, the pipeline fails with the following error in the Event Log: SCHEMA_MISMATCH: Schema of data does not match schema of streaming table: expected 'amount DECIMAL(18,2)' but got 'amount DOUBLE'. The upstream source team has confirmed they changed the amount field type from DECIMAL(18,2) to DOUBLE in a recent data export. The pipeline must resume processing as quickly as possible without data loss. What is the CORRECT remediation sequence?
Answer & explanation
Correct answer: A
WHY A: A SCHEMA_MISMATCH error where an existing column's data type has changed (not just a new column added) requires a schema-breaking change to the streaming table definition. The procedure is: (1) update the streaming table's definition to reflect the new type, (2) perform a Full Refresh (pipeline reset, from the UI or databricks pipelines start --full-refresh), which truncates and rebuilds the target table so the new schema is applied consistently from the beginning, then resume TRIGGERED operation. This is the Lakeflow SDP procedure for type-changing schema evolution.
WHY NOT B: addNewColumns handles the addition of new fields. It does not handle type changes to existing columns, so a DECIMAL-to-DOUBLE change is not resolved by it.
WHY NOT C: Altering the Delta table schema directly via ALTER TABLE outside the pipeline can cause the Lakeflow SDP engine to detect a schema conflict between the pipeline definition and the physical table schema, potentially causing further errors. The pipeline definition and the target table schema are updated in a coordinated way via a Full Refresh.
WHY NOT D: rescue mode captures records that don't match the current schema into _rescued_data. The amount field would land as a JSON string there rather than being processed correctly, and rescued records require manual reprocessing, so this is not a quick resume strategy.
WHY NOT E: Manually deleting the checkpoint directory corrupts the pipeline's incremental state. Databricks manages checkpoints as part of the pipeline metadata, and a Full Refresh through the pipeline UI/API is the supported way to reset pipeline state.