Home / DE Professional practice test / Developing Code for Data Processing using Python and SQL / Question

Databricks Data Engineer Professional · free practice question

A data engineering team has a Lakeflow Spark Declarative Pipeline that ingests from 5 different Auto Loader sources into 5 separate Bronze Streaming Tables…

Exam-style question on Developing Code for Data Processing using Python and SQL from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Developing Code for Data Processing using Python and SQL Medium

A data engineering team has a Lakeflow Spark Declarative Pipeline that ingests from 5 different Auto Loader sources into 5 separate Bronze Streaming Tables, feeds into 3 Silver Streaming Tables (with joins and quality constraints), and produces 2 Gold Materialized Views. The pipeline runs in TRIGGERED mode every hour. During a recent incident, the team discovered that a failure in one of the 5 Auto Loader sources caused the entire pipeline to fail, blocking all Silver and Gold updates even for the unaffected sources. The team wants to improve pipeline resilience so that failures in one Bronze source do not block other independent pipeline branches. Which architectural change BEST addresses this requirement?

  1. AEnable spark.databricks.delta.schema.autoMerge.enabled=true on the pipeline. The engine then merges conflicting schemas across the 5 Bronze sources, which stops schema-related failures from cascading across pipeline branches.
  2. BSplit the pipeline into independent Lakeflow SDP pipelines, one per Bronze source or data domain, each with its own Bronze-to-Gold chain. Orchestrate them in parallel from a Lakeflow Job, which isolates a failure to one pipeline.
  3. CConfigure cloudFiles.allowOverwrites=true on the 5 Auto Loader sources. Auto Loader then keeps processing files from healthy sources when one source errors, which keeps the error out of the shared execution context.
  4. DAdd ON VIOLATION WARN expectations to the Silver Streaming Tables fed by the failing Bronze table. The pipeline then treats a failing Auto Loader source as a constraint violation and continues with the other branches.
  5. ESet pipelines.trigger.interval=5 minutes in the shared pipeline configuration. The engine then runs each Auto Loader source as a separate micro-transaction, which isolates a failure to its micro-batch, not the whole run.
Answer & explanation

Correct answer: B

WHY B: In a single Lakeflow SDP pipeline, the tables share the same pipeline execution context and checkpoint, and a failure in any table definition can halt the entire pipeline update. Splitting into independent pipelines provides true fault isolation: each pipeline has its own checkpoint, its own error boundary, and its own retry/failure state. Job orchestration (Lakeflow Jobs, formerly Databricks Workflows) lets the independent pipelines run in parallel with configurable concurrency, and a failure in one pipeline's run does not affect the others. This is the recommended architectural pattern for resilient multi-source pipelines.

WHY NOT A: delta.schema.autoMerge relates to schema evolution during Delta Lake write operations, not to source-level Auto Loader failures. It does not prevent cascading failures from a failing source.

WHY NOT C: cloudFiles.allowOverwrites controls whether Auto Loader reprocesses files that were overwritten after being processed. It is not related to error isolation between multiple sources.

WHY NOT D: Warn-level expectations handle data quality constraint violations (for example null values or type mismatches in data). A completely failing Auto Loader source (permission error, connectivity issue) is not a constraint violation; it is an infrastructure/source error that halts the streaming read, not a row-level data quality event.

WHY NOT E: pipelines.trigger.interval controls how frequently a flow is triggered. Shorter intervals do not change the error isolation model: the sources still share the same pipeline execution context, and a failure in any source still fails the pipeline run.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →