Home / ML Associate practice test / Data Processing / Question

Databricks Machine Learning Associate · free practice question

A data scientist wants to clean a salary column in a large Spark DataFrame by removing rows where salary is more than 3 standard deviations above or below the…

Exam-style question on Data Processing from the free ML Associate practice test, with the answer and a full explanation below.

Updated November 10, 2026

Data Processing

A data scientist wants to clean a salary column in a large Spark DataFrame by removing rows where salary is more than 3 standard deviations above or below the mean. Which PySpark code correctly implements this 3-sigma outlier removal?

  1. Afrom pyspark.sql import functions as F; stats = df.select(F.mean('salary').alias('mu'), F.stddev('salary').alias('sigma')).collect()[0]; lower = stats.mu - 3 * stats.sigma; upper = stats.mu + 3 * stats.sigma; df_clean = df.filter((F.col('salary') >= lower) & (F.col('salary') <= upper)) — this computes the global mean and standard deviation in a single aggregation job, collects the scalar values, and applies the 3-sigma bounds as a column filter in a subsequent transformation pass
  2. Bfrom pyspark.sql import functions as F; df_stats = df.agg(F.variance('salary').alias('var'), F.mean('salary').alias('mu')); sigma = df_stats.collect()[0].var ** 0.5; df_clean = df.filter(F.abs(F.col('salary') - df_stats.collect()[0].mu) <= 3 * sigma) — using variance is equivalent to stddev since variance equals stddev squared, and F.abs provides the symmetric condition needed for the 3-sigma rule
  3. Cimport pyspark.sql.functions as F; quantiles = df.approxQuantile('salary', [0.0013, 0.9987], 0.05); df_clean = df.filter((F.col('salary') >= quantiles[0]) & (F.col('salary') <= quantiles[1])) — the 0.0013 and 0.9987 percentiles correspond exactly to ±3 standard deviations in a normal distribution, making this percentile-based approach identical to the 3-sigma standard deviation method for all distributions
  4. Dfrom pyspark.ml.feature import StandardScaler; from pyspark.ml.linalg import Vectors; df_vec = df.rdd.map(lambda r: (Vectors.dense(r.salary),)).toDF(['features']); scaler = StandardScaler(inputCol='features', outputCol='scaledFeatures', withMean=True, withStd=True); model = scaler.fit(df_vec); df_scaled = model.transform(df_vec); df_clean = df_scaled.filter(F.abs(df_scaled.scaledFeatures[0]) <= 3.0) — StandardScaler with withMean=True and withStd=True produces true z-scores, and filtering z-scores to [-3, 3] is equivalent to the 3-sigma rule for any distribution
  5. Efrom pyspark.sql import Window, functions as F; window_spec = Window.orderBy('salary'); df_with_z = df.withColumn('z', (F.col('salary') - F.avg('salary').over(window_spec)) / F.stddev('salary').over(window_spec)); df_clean = df_with_z.filter(F.abs(F.col('z')) <= 3) — Window functions compute a running z-score across the salary column ordered by salary value, and filtering z-scores to ±3 correctly identifies and removes global outliers that exceed 3 standard deviations
Answer & explanation

Correct answer: A

WHY A: This is the correct and efficient 3-sigma implementation. It computes the global mean and standard deviation with a single aggregation, collects the scalar bounds, and applies them as a simple column filter — following the standard two-step pattern (aggregate then filter) recommended for outlier removal in PySpark. WHY NOT B: This approach calls .collect() twice on df_stats, triggering two separate Spark jobs and risking data inconsistency; the correct approach collects once and reuses the scalar values. WHY NOT C: The 0.13th and 99.87th percentiles only approximate ±3 sigma on a perfectly normal distribution; this is not equivalent for skewed distributions and is also not the direct standard deviation method as required. WHY NOT D: StandardScaler with withMean=True and withStd=True does produce z-scores, but this approach requires costly RDD-to-DataFrame conversion and vector wrapping, making it unnecessarily complex for a simple column-level outlier filter. WHY NOT E: Using Window.orderBy() creates a running/cumulative window, not a global aggregation; avg() and stddev() over this window compute running statistics row by row, not the global mean and stddev needed for the 3-sigma rule.

Take the full ML Associate practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Machine Learning Associate practice test. Take it and get a section-by-section score.

Start the free test →