A data engineer is writing unit tests for a PySpark transformation function apply_revenue_adjustments(df: DataFrame) -> DataFrame that performs type casting, a column computation, and a filter. The engineer wants to test the function locally without connecting to a Databricks cluster, using pytest. The test should verify both the schema of the output DataFrame and the correctness of specific row values. The engineer also wants to use DataFrame.transform() for composing multiple transformation functions in a testable pipeline chain. Which test implementation correctly uses assertSchemaEqual, assertDataFrameEqual, and DataFrame.transform()?
Answer & explanation
Correct answer: C
WHY C: This is the most complete and correct test using all three specified APIs. df.transform(apply_revenue_adjustments) uses DataFrame.transform() to apply the function in a composable, chain-friendly way. assertDataFrameEqual(result, expected_df, checkRowOrder=False) uses the PySpark testing API to compare DataFrames with configurable row-order sensitivity — essential for non-deterministic query execution. assertSchemaEqual(result.schema, expected_df.schema, ignoreNullable=True) compares schemas while tolerating nullability differences between inferred and explicitly defined schemas. Both assertDataFrameEqual and assertSchemaEqual are from pyspark.testing and provide detailed failure messages on mismatch. WHY NOT A: This test uses assertSchemaEqual and assertDataFrameEqual correctly, but the expected_df has the same number of rows as the result (only 1 row with active status after filtering) — the test logic appears correct but expected_schema is not defined in the test scope shown, making it incomplete. Option C is more complete and correct. WHY NOT B: Using assert result.schema == expected_schema (Python equality) and result.first()['revenue'] are not the idiomatic PySpark testing API. They lack the detailed error messages of assertDataFrameEqual/assertSchemaEqual and don't handle nullable field differences or row ordering. WHY NOT D: apply_revenue_adjustments accepts a DataFrame, not a Python list. This test would fail with a TypeError before any assertions. Unit tests for PySpark functions require a SparkSession fixture. WHY NOT E: While parametrize is a valid pytest pattern and DataFrame.transform() is used correctly, this test only checks a single column value per parametrized case and does not use assertDataFrameEqual or assertSchemaEqual as required by the objective.