A data scientist applies PySpark MLlib's OneHotEncoder to a region column that has 5 distinct category values (indices 0–4 after StringIndexer). The encoder uses the default dropLast=True setting. What is the dimensionality of the resulting sparse vector, and what is the purpose of the dropLast=True default?
Show answer & explanation
Correct answer: E
WHY E: With 5 distinct categories and dropLast=True, the resulting vector has 4 elements. The 5th category is implicitly represented by the all-zeros vector (when all four indicators are 0, the row must belong to the dropped last category). This default is specifically designed to prevent the dummy variable trap, a form of perfect multicollinearity where the sum of all indicators always equals 1, making one column perfectly predictable from the others, which causes problems in linear models such as logistic regression and linear SVM. WHY NOT A: The output has 4 elements, not 5; dropLast=True reduces the vector dimension by one and does not preserve an indicator for each of the 5 categories. It has nothing to do with a slot for unseen labels. WHY NOT C: OHE does not add a bias term; intercept handling in linear models is controlled by the fitIntercept parameter in the model itself, not by OHE. WHY NOT D: dropLast=True drops the last category by index order (the least frequent category, since StringIndexer sorts by descending frequency, placing the most common at index 0 and the least common at the last index); it is not dropping the most frequent category and it is not an overfitting control. WHY NOT B: The vector has 4 elements, not 5; the all-zeros vector represents the dropped last category, not a missing value indicator; missing value handling is controlled by the handleInvalid parameter.