Home / DE Associate practice test / Productionizing Data Pipelines

Free · 7 questions with explanations

Productionizing Data Pipelines: Databricks Data Engineer Associate Practice Questions

Exam-style questions on Productionizing Data Pipelines. Pick your answer, then open the explanation to see why it's right — and why the other options are wrong.

Updated November 10, 2026

1 Productionizing Data Pipelines

A data engineering team wants to ensure their Databricks Asset Bundle deployment process follows CI/CD best practices. They are setting up a GitHub Actions pipeline. In the CI step (on pull request), they want to validate that the bundle configuration is syntactically correct and all referenced files exist, WITHOUT actually deploying any resources to the Databricks workspace. In the CD step (on merge to main), they want to deploy to production. Which pair of databricks bundle commands should be used for the CI and CD steps respectively?

  1. ACI: databricks bundle plan -t prod to generate a deployment plan showing what would change, similar to Terraform plan; CD: databricks bundle apply -t prod to execute the plan and deploy resources, similar to Terraform apply.
  2. BCI: databricks bundle validate to check the bundle configuration for syntax errors, schema compliance, and file reference validity without connecting to or modifying any Databricks workspace; CD: databricks bundle deploy -t prod to authenticate, upload artifacts, and create or update all defined workspace resources in the production target.
  3. CCI: databricks bundle test --dry-run -t prod to simulate a full deployment against a sandbox environment and report any resources that would fail to provision; CD: databricks bundle deploy -t prod --confirm to deploy with an explicit confirmation flag required for production targets.
  4. DCI: databricks bundle lint -t prod to run static analysis checks on all YAML and Python files in the bundle, including checking for deprecated Databricks Runtime versions and unsupported cluster configurations; CD: databricks bundle release -t prod to tag the bundle version in Git and trigger the production deployment workflow.
  5. ECI: databricks bundle check -t prod to verify that all Databricks workspace permissions required by the bundle are correctly configured and that the service principal used for deployment has sufficient access; CD: databricks bundle push -t prod to upload all bundle artifacts to the Databricks workspace file system before a separate bundle activate command makes the resources live.
Show answer & explanation

Correct answer: B

WHY B: databricks bundle validate is the correct command for CI validation—it checks YAML syntax, schema compliance, variable references, and file existence without making any API calls to a Databricks workspace. databricks bundle deploy -t prod is the correct CD command that uploads artifacts, resolves variables for the production target, and creates/updates all defined resources. WHY NOT A: databricks bundle plan and databricks bundle apply are not valid DAB commands. DAB does not use a plan/apply pattern like Terraform. WHY NOT C: databricks bundle test --dry-run and databricks bundle deploy --confirm are not valid commands. There is no dry-run or --confirm flag in the standard DAB CLI. WHY NOT D: databricks bundle lint and databricks bundle release are not valid DAB commands. WHY NOT E: databricks bundle check and databricks bundle push / bundle activate are not valid DAB commands. The correct upload+deploy command is simply databricks bundle deploy.

2 Productionizing Data Pipelines

A data engineer is inspecting the Spark UI for a job that performs a large aggregation. In the 'SQL / DataFrame' tab, the physical plan shows the following operators in sequence: Exchange hashpartitioning → Sort → HashAggregate. The job is taking much longer than expected. The engineer enables Adaptive Query Execution (AQE). Which AQE optimization is MOST relevant to this aggregation pattern, and what specific improvement does it make?

  1. AAQE's Dynamic Partition Pruning is most relevant here. It intercepts the Exchange operator and prunes shuffle partitions that hold no rows matching the aggregation's GROUP BY keys, reducing the data sent across the network ahead of the Sort and HashAggregate operators and shortening the shuffle read for each task.
  2. BAQE's Broadcast Join Conversion is most relevant here. When AQE finds at runtime that the aggregation's input is smaller than the broadcast threshold after the Exchange operator, it converts the HashAggregate into a BroadcastHashAggregate and removes the Sort operator from the physical plan for the rest of the query.
  3. CAQE's Skew Join Optimization is most relevant here. It detects that one or more HashAggregate partitions are much larger than the others and splits each oversized aggregate partition into smaller sub-partitions handled by separate tasks, spreading the final aggregation phase across more of the cluster's executors.
  4. DAQE's Coalescing of Shuffle Partitions is most relevant here. After the Exchange hashpartitioning step, AQE reads the actual sizes of the shuffle output partitions at runtime and merges small ones ahead of the Sort and HashAggregate operators, reducing the task count and the scheduling overhead of many tiny partitions.
  5. EAQE's Runtime Filter Injection is most relevant here. AQE reads the GROUP BY column statistics after the Exchange step and injects a Bloom filter into the HashAggregate operator, skipping rows whose key values were aggregated in a prior micro-batch and reducing the time spent in the final aggregate computation.
Show answer & explanation

Correct answer: D

WHY D: AQE's adaptive coalescing of shuffle partitions is the most impactful optimization for aggregation workloads. The default spark.sql.shuffle.partitions=200 often creates many tiny partitions for small-to-medium datasets. AQE measures actual shuffle map output sizes and merges small adjacent partitions before downstream tasks read them, dramatically reducing task count and scheduling overhead. WHY NOT A: Dynamic Partition Pruning applies to joins with partition filters, not to aggregation pipelines. It prunes data file partitions (such as Delta table partitions), not shuffle partitions. WHY NOT C: Skew Join optimization applies to join operations, not aggregations. While AQE can help with skewed aggregation keys, it does so through the coalescing mechanism, not a dedicated 'skew aggregation' feature that splits aggregate partitions. WHY NOT B: BroadcastHashAggregate is not a standard Spark operator. AQE's broadcast conversion applies to join strategies, not aggregation operators. WHY NOT E: Bloom filter injection for aggregation deduplication across micro-batches is not an AQE feature; AQE operates within a single query's execution, not across multiple runs.

3 Productionizing Data Pipelines

A data engineer is reviewing a Databricks Asset Bundle project that another team member created. The databricks.yml file contains the following snippet: ``yaml targets: dev: mode: development workspace: host: https://adb-dev-1234.azuredatabricks.net variables: catalog_name: dev_catalog prod: mode: production workspace: host: https://adb-prod-5678.azuredatabricks.net variables: catalog_name: prod_catalog ` The engineer wants to deploy to the production environment. Which databricks bundle CLI command is correct, and what behavior will the mode: production` setting enforce?

  1. Adatabricks bundle deploy --target=production is correct, but mode: production has no functional enforcement effect—it is only a metadata label used for documentation purposes within the bundle configuration and does not change any deployment behavior.
  2. Bdatabricks bundle deploy -t prod is the correct command. The mode: production setting enforces behaviors such as preventing concurrent runs of the same job by default, tagging all deployed resources with the bundle name for auditability, and disabling 'development mode' protections like resource name prefixing—ensuring production resources have clean, unprefixed names.
  3. Cdatabricks bundle publish --env prod is the correct command, as DAB uses the publish subcommand for production targets to distinguish them from lower-environment deploy operations, and mode: production automatically triggers a blue-green deployment strategy to avoid downtime.
  4. Ddatabricks bundle deploy --workspace prod is the correct command, as the --workspace flag is used to specify the target name in the YAML configuration, and mode: production automatically locks the bundle configuration to prevent any further edits until the deployment is confirmed as healthy.
  5. Edatabricks bundle deploy --stage prod is the correct command, using the --stage flag to reference the named target, and mode: production automatically enables Delta Lake data retention policies and Z-ordering on all tables written by the deployed pipeline.
Show answer & explanation

Correct answer: B

WHY B: The correct CLI flag for targeting a named environment in DAB is -t or --target followed by the target name defined in the YAML (prod in this case). mode: production enforces several safeguards: disables development-mode resource name prefixing (so resources get clean names like the job name itself, not [dev_username] job_name), prevents concurrent runs by default, and tags resources. WHY NOT A: mode: production is not just a label—it actively changes deployment behavior. WHY NOT C: There is no databricks bundle publish command; deploy is used for all targets. There is also no built-in blue-green deployment strategy. WHY NOT D: The correct flag is --target or -t, not --workspace. There is no deployment lock mechanism in DAB. WHY NOT E: The correct flag is --target or -t, not --stage. DAB does not automatically configure Delta Lake data retention or Z-ordering.

4 Productionizing Data Pipelines

During a Spark UI analysis, a data engineer observes that a job's 'Executors' tab shows several executors with very high 'GC Time' as a percentage of 'Task Time' (over 25%), while other executors show normal GC time (under 5%). All executors have identical hardware configurations. What is the MOST likely cause of this uneven GC pressure, and what investigation step should the engineer take next in the Spark UI?

  1. AThe affected executors are running tasks with Python UDFs, where serialization between the JVM and the Python worker process creates excess JVM object allocation. The next step is to open the 'Environment' tab in the Spark UI and confirm whether spark.python.worker.reuse=false is set, which would spawn a new Python process for each task.
  2. BThe affected executors are co-located on the same physical cloud VM and share CPU resources, which starves the JVM's GC threads of CPU time. The next step is to open the cluster's 'Metrics' tab and compare per-node CPU utilization across the worker nodes to identify the hosts where several executors are competing for the same cores.
  3. CThe affected executors are processing disproportionately large partitions (data skew), which fill the JVM heap with live objects and trigger frequent garbage collection. The next step is to open the 'Stages' tab, select the relevant stage, and compare the task metrics distribution, such as 'Input Size / Records' and 'Shuffle Read Size', for tasks on those executors.
  4. DThe affected executors are running a different JVM build that uses a less efficient garbage collector (for example Serial GC instead of G1GC). The next step is to open the 'Environment' tab in the Spark UI and compare each executor's java.vm.version property and GC flags to identify the executors that are running the older JVM build.
  5. EThe affected executors hold too many cached RDD partitions in the JVM heap through the block manager, which exhausts the heap and forces GC to evict cached blocks. The next step is to open the 'Storage' tab in the Spark UI and review the 'RDD Storage Info' section to see which cached RDDs consume the most memory on those executors.
Show answer & explanation

Correct answer: C

WHY C: Uneven GC pressure across identically configured executors is a strong indicator of data skew. Executors processing larger partitions hold more live JVM objects in their heap, causing more frequent and longer GC cycles. The Stages tab's task-level metrics (input size, shuffle read size) directly reveal the partition size imbalance. WHY NOT A: Python UDFs cause serialization and deserialization overhead on every executor that runs Python tasks, not selectively on a few, so they would not explain an uneven GC distribution. WHY NOT B: Cloud VM scheduling does not co-locate Spark executors from the same application on one VM in a way that creates this pattern; Spark typically assigns one executor per VM (or uses slots within a VM). WHY NOT D: Databricks clusters run a uniform, managed JVM version across their executors; a JVM version mismatch is not possible within a single cluster. WHY NOT E: RDD caching is a valid cause of GC pressure, but it would affect every executor that holds cached partitions rather than a specific few with high GC next to others with low GC.

5 Productionizing Data Pipelines

Given a scenario where a data engineer is evaluating serverless compute for a production Delta Live Tables (DLT) pipeline, which of the following statements about Databricks Serverless compute is ACCURATE?

  1. AServerless compute for DLT pipelines provisions resources in the customer's own cloud account VPC, giving the customer full network visibility and control over egress traffic while Databricks manages the orchestration layer.
  2. BServerless compute requires the customer to pre-purchase a reserved instance commitment with their cloud provider to access the serverless pool, after which Databricks manages the allocation of those reserved instances automatically.
  3. CServerless compute runs in Databricks-managed cloud infrastructure, starts within seconds, automatically scales to match pipeline demands, requires no cluster configuration by the user, and bills only for the compute actually consumed—making it ideal for workloads with unpredictable or bursty resource needs.
  4. DServerless compute for DLT is only available for SQL-based pipelines and does not support Python-defined Delta Live Tables expectations or Python UDF decorators within the pipeline definition.
  5. EServerless compute uses a shared multi-tenant execution environment where multiple customers' pipeline code runs on the same worker nodes simultaneously, which reduces cost but requires customers to avoid storing secrets or PII in pipeline variables.
Show answer & explanation

Correct answer: C

WHY C: This accurately describes Databricks Serverless: it runs in Databricks-managed infrastructure (not the customer's VPC), provisions in seconds, auto-scales, requires zero cluster configuration, and uses a consumption-based billing model. WHY NOT A: Serverless compute runs in Databricks-managed infrastructure, NOT the customer's VPC. The customer gets less network-level control, not more. WHY NOT B: No reserved instance commitment is needed for serverless; it is fully on-demand with no pre-purchase requirement. WHY NOT D: Serverless DLT supports both Python and SQL pipeline definitions, including Python expectations and decorators. WHY NOT E: While Databricks uses multi-tenant infrastructure, there is strict isolation between customer workloads—pipeline code does not co-mingle on the same worker nodes with other customers.

6 Productionizing Data Pipelines

A data engineer notices that a Spark job processing 500 million records is taking 45 minutes to complete, significantly longer than expected. They open the Spark UI to investigate. In the 'Stages' tab, they observe one stage with 200 tasks where 199 tasks completed in under 10 seconds each, but one task took 38 minutes. The data being processed is partitioned by a country_code column. What does this pattern MOST LIKELY indicate, and what is the recommended remediation?

  1. ADriver memory pressure: the driver ran short of memory while collecting results from the 200 tasks, and a long garbage collection pause delayed the acknowledgment of the last task. Remediation: increase driver memory and enable off-heap storage for the result set.
  2. BData skew: one partition, likely a dominant country_code value such as 'US', holds a disproportionate share of the data, and the slow task is processing it. Remediation: salt the skewed key, apply a skew hint, or enable AQE skew join handling (spark.sql.adaptive.skewJoin.enabled=true).
  3. CScheduler misplacement: a YARN resource manager misconfiguration made the scheduler queue the 200 tasks on a single executor, serializing work that was meant to run in parallel. Remediation: fix the YARN node manager memory settings and raise spark.executor.instances.
  4. DShuffle throttling: the shuffle write produced an oversized shuffle file for one country_code partition, and the external shuffle service throttled its transfer to stay within cloud storage bandwidth quotas. Remediation: raise spark.sql.shuffle.partitions.
  5. EBroadcast fallback: the broadcast lookup table exceeded the 8 MB default threshold mid-execution, and Spark fell back to a sort-merge join for one partition while finishing broadcast joins for the rest. Remediation: raise spark.sql.autoBroadcastJoinThreshold.
Show answer & explanation

Correct answer: B

WHY B: This is the classic data-skew symptom: nearly every task finishes quickly, but one straggler task takes orders of magnitude longer. When data is partitioned by country_code, a country such as 'US' might hold 60% or more of the data in a single partition, creating a massive imbalance. AQE's skew join handling, salting, or a skew hint are the standard remediations. WHY NOT A: Driver GC pauses do not show up as a single task taking 38 minutes while the others complete normally; they would affect driver-side operations such as collect(), not one individual task. WHY NOT C: If the tasks had been queued on one executor by a scheduler problem, the slowdown would be spread across the tasks rather than concentrated in one, and Databricks clusters do not use YARN. WHY NOT D: Increasing spark.sql.shuffle.partitions redistributes shuffle output but does not fix skew caused by an inherently imbalanced key distribution, and the shuffle service does not throttle transfers for storage quotas. WHY NOT E: Broadcast join fallback is not a per-partition decision made mid-execution; Spark chooses the join strategy at the plan level before the stage runs.

7 Productionizing Data Pipelines

A data engineer is tasked with deploying a production Databricks Workflow that orchestrates four tasks: (1) a Python notebook for data ingestion, (2) a Delta Live Tables pipeline for transformation, (3) a SQL notebook for data quality checks, and (4) a Python script for sending an email notification on success or failure. The team uses Databricks Asset Bundles. The engineer also needs to configure the workflow so that Task 4 (notification) always runs, even if Tasks 2 or 3 fail. In the DAB job resource YAML, which task dependency configuration correctly implements the 'always run notification' requirement?

  1. ASet Task 4's depends_on to reference all three upstream tasks with outcome: success for each dependency. This instructs Databricks to run Task 4 only after confirming all upstream tasks completed successfully, which is the only supported dependency outcome type in DAB YAML configurations.
  2. BSet Task 4's depends_on to list Tasks 1, 2, and 3 without any outcome filter (or with the default outcome: success). Then, in the Workflows UI after deployment, manually edit Task 4's dependency settings to add an 'On failure' trigger path, since DAB YAML does not support conditional dependency outcomes and this configuration must be done through the UI.
  3. CDefine Task 4 as a completely independent task with no depends_on configuration, and instead configure it with a separate trigger schedule that fires 10 minutes after the main workflow's expected completion time, ensuring it runs regardless of upstream task outcomes.
  4. DIn the DAB job resource YAML, configure Task 4's depends_on block to reference each upstream task with outcome: success_or_failed for Tasks 2 and 3 (to handle both success and failure), and outcome: success for Task 1. This combination of outcome types within a single task's dependency block ensures Task 4 runs after all upstream tasks settle, regardless of their individual outcomes.
  5. EIn the DAB job resource YAML, configure Task 4's dependency block to reference Tasks 1, 2, and 3. For each dependency where Task 4 should run regardless of outcome, set the run_if condition at the task level (e.g., run_if: ALL_DONE) so that Task 4 executes after all its upstream dependencies have reached a terminal state (success, failure, or skipped), while not requiring all of them to succeed.
Show answer & explanation

Correct answer: E

WHY E: Databricks Workflows supports a run_if condition at the task level that controls when a task runs relative to its dependencies. ALL_DONE means the task runs after all upstream dependencies reach a terminal state regardless of their success or failure status. This is the correct DAB YAML configuration for implementing 'always run' notification tasks. Other valid run_if values include ALL_SUCCESS (default), AT_LEAST_ONE_SUCCESS, NONE_FAILED, ALL_FAILED, AT_LEAST_ONE_FAILED. WHY NOT A: outcome: success on all dependencies means Task 4 only runs if ALL upstream tasks succeed—it would not run if Tasks 2 or 3 fail. WHY NOT B: DAB YAML fully supports conditional dependency/run_if configurations; a UI workaround is unnecessary and brittle. WHY NOT C: Using a separate schedule with a time offset is unreliable (job duration varies) and creates a separate workflow, not the required conditional dependency within the same workflow. WHY NOT D: outcome: success_or_failed is not a valid dependency outcome type in Databricks Workflows YAML. The correct mechanism is the task-level run_if condition.

Take the full DE Associate practice test →

From the free full-length test

16 more Productionizing Data Pipelines questions with answers

Each opens on its own page with the correct answer and the full explanation.

All free Data Engineer Associate exam questions & answers →