A team's PySpark transformation logic is factored into pure functions. On every pull request, a CI runner must execute a pytest suite against those functions quickly and cheaply, without provisioning any Databricks cluster. Which approach best fits this CI step?
Answer & explanation
Correct answer: E
WHY E: Pure PySpark transformation functions can be exercised with pytest on the CI runner via Databricks Connect or a local SparkSession, needing no Databricks cluster for the test step. WHY NOT A: unit tests of pure functions do not require attaching to a live all-purpose cluster; a local Spark session suffices. WHY NOT B: deploying to a staging workspace first is an integration step, not a prerequisite for fast local unit tests. WHY NOT C: transformation logic factored into functions is unit-testable locally, so the claim that it cannot be is false. WHY NOT D: provisioning a job cluster per PR is slow and unnecessary because the tests run against a local Spark session.