A downstream pipeline needs to process incremental changes from an upstream Delta table. The upstream table is NOT append-only: it can receive UPDATE, MERGE, and DELETE operations. The downstream requirement is to capture inserts, updates, and deletes reliably without full reprocessing. Which approach is the most robust?
Answer & explanation
Correct answer: A
WHY A: When the source Delta table includes updates and deletes, streaming from the Change Data Feed is the robust pattern because it represents changes explicitly and supports downstream processing of each change type (insert, update, delete).
WHY NOT B: Streaming from a Delta table itself is recommended when the source is append-only; updates and deletes cause exceptions or require special handling, and ACID commits do not make the stream propagate them.
WHY NOT C: skipChangeCommits ignores commits that modify or delete existing records; it does not propagate those changes downstream.
WHY NOT D: ignoreDeletes is a legacy option that handles one specific pattern (full partition drops) and does not provide reliable propagation of update/delete semantics.