Home / ML Associate practice test / Data Processing

Free · 8 questions with explanations

Data Processing: Databricks Machine Learning Associate Practice Questions

Exam-style questions on Data Processing. Pick your answer, then open the explanation to see why it's right — and why the other options are wrong.

Updated November 10, 2026

1 Data Processing

A data scientist applies PySpark MLlib's OneHotEncoder to a region column that has 5 distinct category values (indices 0–4 after StringIndexer). The encoder uses the default dropLast=True setting. What is the dimensionality of the resulting sparse vector, and what is the purpose of the dropLast=True default?

  1. AThe vector has 5 elements, one per category; dropLast=True removes the trailing empty slot that StringIndexer reserves for unseen labels, keeping a dedicated binary indicator for each of the 5 known regions
  2. BThe vector has 5 elements, one more than the standard 4-element encoding; dropLast=True swaps the last indicator for a null-indicator element that represents rows with a missing region value during inference
  3. CThe vector has 6 elements, one more than the number of categories; dropLast=True appends a built-in bias element after the indicators, letting linear models fit an intercept without the fitIntercept parameter
  4. DThe vector has 4 elements, one fewer than the number of categories; dropLast=True removes the indicator of the most frequent category, pushing the model to learn from rarer regions and reducing overfitting
  5. EThe vector has 4 elements, one fewer than the number of categories; dropLast=True removes the last indicator, which is redundant when the 5 indicators sum to 1, avoiding perfect multicollinearity in linear models
Show answer & explanation

Correct answer: E

WHY E: With 5 distinct categories and dropLast=True, the resulting vector has 4 elements. The 5th category is implicitly represented by the all-zeros vector (when all four indicators are 0, the row must belong to the dropped last category). This default is specifically designed to prevent the dummy variable trap, a form of perfect multicollinearity where the sum of all indicators always equals 1, making one column perfectly predictable from the others, which causes problems in linear models such as logistic regression and linear SVM. WHY NOT A: The output has 4 elements, not 5; dropLast=True reduces the vector dimension by one and does not preserve an indicator for each of the 5 categories. It has nothing to do with a slot for unseen labels. WHY NOT C: OHE does not add a bias term; intercept handling in linear models is controlled by the fitIntercept parameter in the model itself, not by OHE. WHY NOT D: dropLast=True drops the last category by index order (the least frequent category, since StringIndexer sorts by descending frequency, placing the most common at index 0 and the least common at the last index); it is not dropping the most frequent category and it is not an overfitting control. WHY NOT B: The vector has 4 elements, not 5; the all-zeros vector represents the dropped last category, not a missing value indicator; missing value handling is controlled by the handleInvalid parameter.

2 Data Processing

A data scientist is preparing features for a linear regression model. The dataset contains a subscription_tier column with 3 nominal categories: 'Bronze', 'Silver', 'Gold'. There is no natural ordering among these tiers for this model's purpose. Which encoding strategy is most appropriate, and why?

  1. AUse ordinal integer encoding with Bronze=1, Silver=2, Gold=3, because linear regression reads the values as evenly spaced levels and learns one coefficient that scales with tier rank without adding redundant indicator columns
  2. BSkip encoding and pass the raw string column to the model, because Spark MLlib's LinearRegression converts string feature values to numeric TF-IDF weighted embeddings before fitting and manual encoding adds nothing
  3. CUse target (mean) encoding, because it outperforms one-hot encoding for linear models on nominal features with fewer than 10 categories and captures the label signal in one numeric column without added dimensions
  4. DUse StringIndexer alone without OneHotEncoder, because the indices 0, 1, and 2 are close enough in magnitude to binary indicators that the model learns a separate effect for each tier without the full transformation
  5. EUse StringIndexer followed by OneHotEncoder with dropLast=True to get 2 binary indicator columns, because integer codes for unordered tiers would impose a false linear ordering that the model would learn as a real gradient
Show answer & explanation

Correct answer: E

WHY E: One-hot encoding is the appropriate strategy for nominal categorical features in linear regression. Integer encoding (0, 1, 2) would make the model learn a single slope implying that Bronze to Silver to Gold is a uniform linear increase, a false ordinal assumption. OHE creates independent binary indicators, so the model learns a separate additive effect for each tier. With 3 categories and dropLast=True, 2 binary columns are produced, which is sufficient to represent all 3 tiers. WHY NOT A: Assigning 1/2/3 to nominal categories introduces a false ordinal relationship; linear regression treats the numeric gap between categories as a meaningful slope, which is incorrect for unordered data. WHY NOT B: Spark MLlib's LinearRegression requires all features to be numeric; passing a string column raises a runtime error, and there is no internal string-to-embedding conversion. WHY NOT C: Target encoding can be effective, but the blanket claim that it outperforms OHE for linear models is false; it can introduce data leakage if not implemented with cross-fitting, and OHE is the standard reliable approach. WHY NOT D: StringIndexer alone introduces the same false ordinal assumption; even with 3 categories the model treats the codes 0, 1, and 2 as numerically meaningful and learns incorrect coefficient magnitudes.

3 Data Processing

A data scientist wants to understand the relationship between age (continuous, range 18–85) and annual_income (continuous, range $20K–$500K). They want to visualize the relationship and also quantify the strength and direction of any linear association. Which approach is most appropriate?

  1. ACreate separate histograms for age and annual_income using different colors on the same plot, and compute the mean and standard deviation of each feature independently — overlapping histograms reveal the degree of linear correlation between two continuous variables by showing where their individual distributions intersect
  2. BApply a Chi-Square test of independence after binning both age and annual_income into decile groups — binning both continuous variables and applying the Chi-Square test is the most statistically rigorous method for quantifying linear correlation between two continuous numeric features
  3. CCreate a scatter plot with age on the x-axis and annual_income on the y-axis using df.plot.scatter(x='age', y='annual_income'), and compute the Pearson correlation coefficient with df.stat.corr('age', 'annual_income') — the scatter plot reveals the visual pattern and direction of the relationship while the Pearson coefficient quantifies its strength
  4. DApply a two-sample t-test comparing the mean annual_income between observations where age is above the median versus below the median — splitting one continuous variable at the median and applying a t-test is a complete and rigorous method for measuring the linear relationship between two continuous numeric features
  5. ECreate a box plot of annual_income grouped by quartile bins of age — grouping the income values into four age buckets and displaying box plots for each bucket provides a complete measure of the linear correlation between the two continuous variables without requiring any additional statistical tests
Show answer & explanation

Correct answer: C

WHY C: A scatter plot is the standard bivariate visualization for two continuous variables — it reveals the pattern, direction, and strength of the relationship visually. Pearson correlation (df.stat.corr()) provides the numeric coefficient ranging from -1 to +1 quantifying the linear association. Together these two tools are the correct and complete approach for comparing two continuous features. WHY NOT A: Overlapping histograms show individual univariate distributions for each variable separately but cannot reveal the relationship or correlation between the two variables; they are not bivariate analysis tools. WHY NOT B: Chi-Square tests are designed for categorical data; binning continuous variables discards information and the result measures general statistical dependence, not specifically linear correlation — Pearson correlation is the correct test for linear continuous-to-continuous relationships. WHY NOT D: A median-split t-test compares group means but loses all information about how income varies across the full continuous range of age values; it cannot capture linear trends and is a weaker analytical approach. WHY NOT E: While box plots grouped by age bins provide useful directional insight, binning age discards information and the result does not measure linear correlation; it cannot replace the Pearson coefficient for quantifying linear association.

4 Data Processing

A data analyst wants to determine whether there is a statistically significant association between two nominal categorical variables: customer_segment (values: Premium, Standard, Basic) and preferred_channel (values: Mobile, Web, In-Store). Which approach is most appropriate for comparing these two categorical features?

  1. ABuild a contingency table using df.stat.crosstab('customer_segment', 'preferred_channel') to compute the frequency of each segment-channel combination, then apply a Chi-Square test of independence — this is the standard method for testing whether two nominal categorical variables are statistically associated with each other
  2. BCompute the Pearson correlation coefficient between numerically label-encoded values of customer_segment and preferred_channel using df.stat.corr() — Pearson correlation is the standard statistical measure for quantifying the strength and direction of the linear relationship between two categorical variables when they are represented as integers
  3. CCreate a scatter plot with customer_segment on the x-axis and preferred_channel on the y-axis — scatter plots reveal the joint distribution and strength of association between two nominal categorical variables and are the most commonly used comparison method in exploratory data analysis on string-valued features
  4. DCompute the mean of a label-encoded preferred_channel grouped by customer_segment using df.groupBy('customer_segment').agg(F.mean('preferred_channel')) — averaging the label-encoded channel values per segment gives an interpretable numeric measure of which channel each segment prefers most
  5. EApply an independent samples t-test comparing the distribution of a numerically encoded preferred_channel across different levels of customer_segment — the two-sample t-test is the appropriate hypothesis test for comparing two nominal categorical distributions when each category group can be treated as an independent numeric sample
Show answer & explanation

Correct answer: A

WHY A: Cross-tabulation followed by a Chi-Square test of independence is the statistically correct method for testing whether two nominal categorical variables are related. PySpark's df.stat.crosstab() efficiently computes the contingency table, and PySpark's ChiSquareTest class can test for independence. WHY NOT B: Pearson correlation measures linear relationships between continuous numeric variables; applying it to arbitrarily label-encoded categories produces results that depend on the arbitrary encoding order and carry no statistical meaning. WHY NOT C: Scatter plots are designed for two continuous variables; plotting two categorical axes creates a grid of overlapping points that reveals no distributional differences or statistical association. WHY NOT D: Averaging a label-encoded nominal variable is statistically invalid because integer codes assigned to categories have no inherent numeric meaning or ordering — the result is uninterpretable. WHY NOT E: The t-test is designed to compare the means of a continuous numeric variable between two groups, not to test association between two nominal categorical variables; applying it here would be methodologically incorrect.

5 Data Processing

A machine learning engineer is performing exploratory data analysis on a house_price column, which is a continuous numeric feature. They want to understand the shape of its distribution — specifically whether it is approximately normal, right-skewed, or multimodal. Which visualization is most appropriate?

  1. AA bar chart of the 20 most frequent house_price values, for example from df.groupBy('house_price').count() — ranking the commonest price points as categories exposes skew and multiple modes
  2. BA pie chart showing each unique house_price value as a share of the dataset — the relative size of each slice exposes which price points dominate and whether the distribution has several modes
  3. CA scatter plot of house_price against the DataFrame row index — plotting each price in sequence exposes the spread of the values and whether the distribution is skewed or has several modes
  4. DA box-and-whisker plot of house_price with no histogram alongside — the median, interquartile range, and whisker bounds expose skew, outlier tails, and multiple modes in the distribution
  5. EA histogram that groups house_price into bins and shows the count per bin, for example df.plot.hist(column='house_price', bins=30) — adjacent frequency bins expose skew, tails, and multiple modes
Show answer & explanation

Correct answer: E

WHY E: Histograms are the standard and most appropriate visualization for continuous features. By dividing the value range into bins and showing the count per bin, a histogram directly reveals whether the distribution is symmetric, skewed, or multimodal. PySpark's native .plot.hist() (available in Databricks Runtime 17.0+ / Spark 4.0+) and pandas .hist(), for example df_pd['house_price'].hist(bins=30) after converting to Pandas, both support this. WHY NOT A: A bar chart of the top 20 most frequent values ignores the vast majority of the distribution and cannot reveal overall shape, skewness, or multimodality. WHY NOT B: Pie charts are inappropriate for high-cardinality continuous data; with thousands of unique price values, every slice is invisibly thin and no distributional insight can be gained. WHY NOT C: A scatter plot against the row index reflects data entry order, not the statistical distribution shape; it cannot reliably detect skewness or modes. WHY NOT D: While box plots usefully summarize the five-number summary and flag outliers, they cannot detect multimodality (multiple peaks) and do not show the full shape of the distribution; a histogram is necessary for that purpose.

6 Data Processing

A data scientist is exploring a dataset with a product_category column containing 8 distinct nominal string values. She wants to understand the frequency distribution of each category. Which visualization type is most appropriate for this column?

  1. AA line chart connecting the frequency count of each category in alphabetical order; line charts reveal trends across discrete categories and are the standard way to show nominal frequency distributions
  2. BA scatter plot with product_category on the x-axis and row index on the y-axis; the density of plotted points shows where each category appears and reveals unequal frequencies across the categories
  3. CA histogram with default bin selection; histograms detect nominal string columns and group them into equal-width bins by alphabetical order, which suits both categorical and continuous distributions
  4. DA bar chart with category labels on the x-axis and the count of each category on the y-axis, built from df.groupBy('product_category').count(); each bar height shows that category's frequency directly
  5. EA box plot grouped by product_category; box plots display the five-number summary (min, Q1, median, Q3, max) for each group and are the standard chart for the frequency counts of a nominal column
Show answer & explanation

Correct answer: D

WHY D: A bar chart is the standard and most appropriate visualization for nominal categorical data, showing the frequency count of each category as a proportional bar that is easy to compare at a glance. Both Databricks' display() bar chart visualization and df.groupBy('product_category').count().toPandas().plot.bar() support this directly after the groupBy().count() aggregation. WHY NOT A: Line charts imply a continuous trend between adjacent points; connecting nominal categories with a line incorrectly suggests an ordered relationship between them. WHY NOT B: A scatter plot of category against row index produces a cloud that does not summarize frequencies and cannot clearly communicate the relative prevalence of each category. WHY NOT C: Histograms divide a numeric range into bins; they cannot process string columns and are designed for continuous numeric distributions, not nominal data. WHY NOT E: Box plots summarize the distribution of a numeric variable across groups; they are not used to show the frequency counts of a categorical column itself.

7 Data Processing

A data scientist is building a PySpark MLlib preprocessing Pipeline for a dataset with three columns: annual_salary (double), years_experience (integer), and job_title (string). They plan to use PySpark's Imputer class to handle missing values across all three columns in one step. What is an important limitation they must account for?

  1. AThe Imputer supports the 'mean' strategy alone, so median or mode values have to be computed with approxQuantile() or groupBy().agg() and applied with fillna() before the Pipeline is built
  2. BThe Imputer treats Python None as missing but skips NaN values in double columns, so NaN entries in annual_salary have to be converted to None with F.when() before the DataFrame is passed in
  3. CThe Imputer supports numeric types such as integer and double but not string or categorical columns, so job_title has to be handled separately with a different imputation approach
  4. DThe Imputer is a standalone utility that is not a valid Pipeline stage, so it has to be fitted and applied outside the Pipeline, and listing it in stages raises an error during Pipeline.fit()
  5. EThe Imputer casts double columns to integer when the computed surrogate is a whole number, so decimal values in annual_salary are truncated and the column loses precision after imputation
Show answer & explanation

Correct answer: C

WHY C: The PySpark Imputer documentation explicitly states: 'The imputer does not support categorical features and possibly creates incorrect values for a categorical feature.' The job_title string column cannot be passed to the Imputer; it must be handled separately (for example by filling with the most frequent title using a custom approach or SQL). WHY NOT A: PySpark's Imputer supports 'mean', 'median', and 'mode' strategies; all three are documented and implemented, so no manual computation is needed. WHY NOT B: By default the Imputer treats both NaN and null/None as missing for float/double columns (the missingValue parameter defaults to NaN); it does not skip NaN values. WHY NOT D: The Imputer is a standard MLlib Estimator and is fully compatible with Pipeline stages; it is commonly chained with StringIndexer, OneHotEncoder, and VectorAssembler. WHY NOT E: The Imputer preserves the original data type of its input columns in the output; it does not cast between double and integer during imputation.

8 Data Processing

A data scientist wants to explore a large Databricks Spark DataFrame to understand numeric distributions, string column frequencies, and see inline visual histograms of value distributions — all in a single command without writing extra plotting code. Which tool best satisfies all of these requirements inside a Databricks notebook?

  1. ACall df.summary() and then pipe the resulting statistics DataFrame to matplotlib — df.summary() returns all statistical metrics and automatically renders distribution histograms for each column when its output is passed to matplotlib.show() in a Databricks notebook cell
  2. BCall df.describe() — this PySpark method computes count, mean, stddev, min, and max for numeric and string columns and also renders embedded histograms alongside each metric when executed in a Databricks notebook environment
  3. CUse df.stat.freqItems(['col1', 'col2']) — this method computes frequency distributions across all specified columns and automatically renders interactive visualizations for both numeric and categorical features directly in the Databricks notebook output cell
  4. DCall dbutils.data.summarize(df) — this Databricks utility generates a rich data profile for the entire DataFrame including summary statistics, distinct value counts, and inline histograms of value distributions for numeric, string, and date columns, all rendered directly in the notebook
  5. EConvert the Spark DataFrame to Pandas with .toPandas() and then call pandas_df.describe() — this produces the same comprehensive data profile as dbutils.data.summarize() and automatically renders inline histogram visualizations for each column without requiring additional imports or plotting code
Show answer & explanation

Correct answer: D

WHY D: dbutils.data.summarize(df) is the Databricks-specific tool designed to render a full interactive data profile — including summary statistics, null counts, distinct value distributions, and visual histograms for numeric, string, and date columns — in a single call within a Databricks notebook. WHY NOT A: df.summary() returns a statistical summary as a Spark DataFrame but does not automatically render histograms; additional plotting code is required. WHY NOT B: df.describe() provides basic statistics (count, mean, stddev, min, max) but does not render histograms. WHY NOT C: df.stat.freqItems() computes only the most frequent items for specified columns and does not produce a full visual data profile. WHY NOT E: While .toPandas().describe() provides numeric statistics, it does not render histograms automatically and risks out-of-memory errors on large Spark DataFrames; it cannot replicate the inline visual profile produced by dbutils.data.summarize().

Take the full ML Associate practice test →