A data engineer needs to remove outliers from a transaction_amount column in a Spark DataFrame using the Interquartile Range (IQR) method — specifically, keeping only rows where transaction_amount falls within [Q1 − 1.5 × IQR, Q3 + 1.5 × IQR]. Which PySpark code correctly implements this?
Answer & explanation
Correct answer: B
WHY B: Using percentile_approx via expr() is the idiomatic PySpark approach for IQR-based outlier removal — it computes Q1 and Q3 efficiently in a single aggregation, derives the 1.5×IQR fence, and applies it as a filter. WHY NOT A: StandardScaler produces standard-deviation-based z-scores, not IQR fences; filtering scaled values to [-1.5, 1.5] implements the 1.5-sigma rule, not the IQR method. WHY NOT C: The .summary() method returns percentile rows as strings in a transposed schema; the indexing logic shown is incorrect and percentile values are not returned as floats without explicit casting from the named row structure. WHY NOT D: This uses mean ± 1.5 × stddev, which is the z-score/standard deviation method — a completely different technique from the IQR method and not equivalent on skewed distributions. WHY NOT E: Calling approxQuantile() three times inline inside a filter clause triggers three separate Spark jobs and is highly inefficient; this approach is functionally fragile and not the recommended pattern for IQR outlier removal.