A Databricks production Delta table has deletion vectors enabled (delta.enableDeletionVectors = true). The operations team notices that after multiple DELETE and UPDATE operations, the underlying Parquet files have NOT been rewritten and old modified row data still physically exists in storage. They need to guarantee all rows marked by deletion vectors are physically purged from Parquet files for a compliance data retention requirement. Which statement correctly explains the behavior of deletion vectors and the correct command to physically purge all rows marked by deletion vectors?
Answer & explanation
Correct answer: A
WHY A: Deletion vectors are a storage optimization that performs soft-deletes: DELETE, UPDATE, and MERGE operations mark rows as modified in a deletion vector sidecar file without rewriting the underlying Parquet data. Reads resolve current state by applying these modifications at query time. Physical changes to Parquet files occur when (1) OPTIMIZE runs (incremental compaction), (2) auto-compaction triggers, or (3) REORG TABLE ... APPLY (PURGE) is explicitly run. For a compliance purge, REORG TABLE ... APPLY (PURGE) is the command: it rewrites the data files containing records with deletion vector modifications. After REORG completes, VACUUM is run to remove the old unreferenced files.
WHY NOT B: Deletion vectors purposely avoid rewriting Parquet files during DELETE/UPDATE; that is the performance benefit, so leftover files are expected and not a checkpointing failure. OPTIMIZE with a WHERE clause does incremental clustering/compaction but does not ensure that the deletion-vector-marked rows are physically purged.
WHY NOT C: VACUUM removes unreferenced data files after the retention period; it does NOT rewrite files that still contain deletion-vector-marked rows. Files still referenced by the current table version are left untouched.
WHY NOT D: DROP FEATURE removes the deletion vector capability from the table protocol; it is a protocol downgrade operation, not the data purge mechanism, and it does not remove the old data files. The compliance approach is REORG TABLE ... APPLY (PURGE) followed by VACUUM.
WHY NOT E: Deletion vectors apply to DELETE, UPDATE, and MERGE operations, not just DELETE. UPDATE with deletion vectors enabled also uses soft-deletes rather than rewriting files at write time (especially with Photon for MERGE and UPDATE), and OPTIMIZE FULL is not the purge command.