Home / DE Professional practice test / Developing Code for Data Processing using Python and SQL / Question

Databricks Data Engineer Professional · free practice question

A data engineering team has an existing production Spark Structured Streaming job written in PySpark that reads from Delta Lake, applies a stateful…

Exam-style question on Developing Code for Data Processing using Python and SQL from the free DE Professional practice test, with the answer and a full explanation below.

Updated November 16, 2026

Developing Code for Data Processing using Python and SQL Medium

A data engineering team has an existing production Spark Structured Streaming job written in PySpark that reads from Delta Lake, applies a stateful sessionization window function using flatMapGroupsWithState, and writes results to a Gold Delta table. The team is evaluating migrating this job to Lakeflow Spark Declarative Pipelines. A senior engineer raises concerns about whether Lakeflow SDP can support all the features required. Which statement MOST accurately describes the capabilities and limitations of Lakeflow SDP compared to the team's direct Structured Streaming job for this specific use case?

  1. ALakeflow SDP does not natively support flatMapGroupsWithState or other arbitrary stateful operators in its declarative table definitions; it targets SQL-expressible transformations and common streaming patterns. A direct Structured Streaming job remains the better fit for this custom stateful logic.
  2. BLakeflow SDP can replace the job, as it runs pipeline code on the same Structured Streaming engine. flatMapGroupsWithState can be applied inside a @dlt.table function to the streaming DataFrame before it is returned, and the framework accepts arbitrary streaming operators there.
  3. CLakeflow SDP is more capable than direct Structured Streaming, as it runs on a newer proprietary Databricks engine that replaced Structured Streaming internally. flatMapGroupsWithState and the other operators run there with better performance, and migration is recommended.
  4. DThe team can migrate by declaring the Gold table as a Streaming Table and using APPLY CHANGES INTO with a SEQUENCE BY expression that reproduces the sessionization logic. SEQUENCE BY accepts arbitrary windowing functions, which makes it a replacement for flatMapGroupsWithState.
  5. EBoth approaches are equivalent in capability, but direct Structured Streaming is preferable in production: the pipeline metadata, event logging, and Unity Catalog lineage in Lakeflow SDP add 15–30% query latency compared with an equivalent direct job running the same transformations.
Answer & explanation

Correct answer: A

WHY A: Lakeflow SDP is built on Spark Structured Streaming and handles the vast majority of streaming ETL patterns (APPLY CHANGES / AUTO CDC, aggregations, joins) extremely well, but it is a declarative framework that constrains table definitions to SQL-expressible or DataFrame API patterns that fit its table update model. flatMapGroupsWithState is a low-level stateful operator that manages arbitrary per-key state across micro-batches using a custom user-defined state transition function. This type of arbitrary stateful processing does not map cleanly to Streaming Table or Materialized View constructs. For such cases, direct Structured Streaming (the DataFrame API in a regular Python/Scala job) gives the team full access to every streaming operator. The guidance is to use Lakeflow SDP for the majority of pipeline logic and fall back to direct Structured Streaming when custom stateful operators are required.

WHY NOT B: @dlt.table functions do work with a streaming DataFrame, but the framework imposes constraints on which operators can be applied: arbitrary stateful operators like flatMapGroupsWithState are not supported within the declarative pipeline execution model.

WHY NOT C: Lakeflow SDP is not a replacement streaming engine; it runs on top of Spark Structured Streaming. It does not replace Structured Streaming internally, and flatMapGroupsWithState is not available within pipeline table definitions.

WHY NOT D: APPLY CHANGES INTO (AUTO CDC) is for CDC (insert/update/delete from a change stream), not for arbitrary sessionization window logic. The SEQUENCE BY clause determines event ordering for CDC operations, not custom stateful window computations.

WHY NOT E: Lakeflow SDP's management overhead does not add 15–30% query latency. The metadata and lineage operations are asynchronous and do not block query execution paths, and the two approaches are not equivalent in capability.

Take the full DE Professional practice test free →

Would you pass today?

This question is one of 60 in the free, timed Databricks Data Engineer Professional practice test. Take it and get a section-by-section score.

Start the free test →