Home / ML Associate practice test / Data Processing / Question

Databricks Machine Learning Associate · free practice question

A data engineer needs to remove outliers from a transaction_amount column in a Spark DataFrame using the Interquartile Range (IQR) method — specifically…

Exam-style question on Data Processing from the free ML Associate practice test, with the answer and a full explanation below.

Updated November 10, 2026

Data Processing

A data engineer needs to remove outliers from a transaction_amount column in a Spark DataFrame using the Interquartile Range (IQR) method — specifically, keeping only rows where transaction_amount falls within [Q1 − 1.5 × IQR, Q3 + 1.5 × IQR]. Which PySpark code correctly implements this?

  1. Afrom pyspark.ml.feature import StandardScaler — StandardScaler normalizes the transaction_amount column so that its z-scores fall between -1 and 1, and filtering the scaled output to the range [-1.5, 1.5] is equivalent to applying the 1.5×IQR Tukey fence on the original distribution
  2. Bfrom pyspark.sql.functions import expr, col; quantiles = df.select(expr('percentile_approx(transaction_amount, 0.25)').alias('Q1'), expr('percentile_approx(transaction_amount, 0.75)').alias('Q3')).collect()[0]; iqr = quantiles.Q3 - quantiles.Q1; df_clean = df.filter((col('transaction_amount') >= quantiles.Q1 - 1.5*iqr) & (col('transaction_amount') <= quantiles.Q3 + 1.5*iqr)) — this correctly uses percentile_approx to compute Q1 and Q3, derives the IQR fence bounds, and filters the DataFrame in a single subsequent transformation
  3. Csummary = df.summary('25%', '75%').collect(); q1 = float(summary[0]['transaction_amount']); q3 = float(summary[1]['transaction_amount']); iqr = q3 - q1; df_clean = df.filter((df.transaction_amount >= q1 - 1.5*iqr) & (df.transaction_amount <= q3 + 1.5*iqr)) — the .summary() method's percentile rows are indexed 0 and 1 in the collected list and return floats that can be directly used in filter conditions
  4. Dfrom pyspark.sql import functions as F; stats = df.select(F.mean('transaction_amount').alias('mu'), F.stddev('transaction_amount').alias('sigma')).collect()[0]; df_clean = df.filter((F.col('transaction_amount') >= stats.mu - 1.5*stats.sigma) & (F.col('transaction_amount') <= stats.mu + 1.5*stats.sigma)) — this implements the IQR method by using mean ± 1.5 × standard deviation, which is mathematically equivalent to the Q1/Q3 fence on symmetric distributions
  5. Edf_clean = df.filter(df.transaction_amount.between(df.approxQuantile('transaction_amount', [0.25], 0.0)[0] - 1.5 * (df.approxQuantile('transaction_amount', [0.75], 0.0)[0] - df.approxQuantile('transaction_amount', [0.25], 0.0)[0]), df.approxQuantile('transaction_amount', [0.75], 0.0)[0] + 1.5 * (df.approxQuantile('transaction_amount', [0.75], 0.0)[0] - df.approxQuantile('transaction_amount', [0.25], 0.0)[0]))) — calling approxQuantile inline inside a filter clause is the most concise pattern and avoids any intermediate variable assignments
Answer & explanation

Correct answer: B

WHY B: Using percentile_approx via expr() is the idiomatic PySpark approach for IQR-based outlier removal — it computes Q1 and Q3 efficiently in a single aggregation, derives the 1.5×IQR fence, and applies it as a filter. WHY NOT A: StandardScaler produces standard-deviation-based z-scores, not IQR fences; filtering scaled values to [-1.5, 1.5] implements the 1.5-sigma rule, not the IQR method. WHY NOT C: The .summary() method returns percentile rows as strings in a transposed schema; the indexing logic shown is incorrect and percentile values are not returned as floats without explicit casting from the named row structure. WHY NOT D: This uses mean ± 1.5 × stddev, which is the z-score/standard deviation method — a completely different technique from the IQR method and not equivalent on skewed distributions. WHY NOT E: Calling approxQuantile() three times inline inside a filter clause triggers three separate Spark jobs and is highly inefficient; this approach is functionally fragile and not the recommended pattern for IQR outlier removal.

Take the full ML Associate practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Machine Learning Associate practice test. Take it and get a section-by-section score.

Start the free test →