A data engineering team is building a production pipeline that ingests raw JSON event files landing continuously in an Azure Data Lake Storage Gen2 container. The pipeline must: (1) detect new files as they arrive with sub-minute latency, (2) handle schema evolution gracefully when new fields are added to the JSON payload, (3) track which files have been processed to avoid reprocessing on pipeline restart, (4) scale to handle bursts of thousands of files arriving within minutes. The team is choosing between configuring Auto Loader with file notification mode vs. directory listing mode. Which configuration correctly addresses all four requirements?
Answer & explanation
Correct answer: C
WHY C: File notification mode with Azure Event Grid + Azure Queue Storage is the correct choice for sub-minute latency at scale. Notifications are pushed in near-real-time as files land, unlike directory listing which must periodically scan the entire prefix. addNewColumns schema evolution mode automatically extends the inferred schema when new JSON fields appear, satisfying requirement 2. Auto Loader's streaming checkpoint durably records processed file offsets, preventing reprocessing on restart (requirement 3). Event Grid + Queue Storage scales to handle thousands of events per second, satisfying requirement 4. WHY NOT A: Directory listing mode scans the directory on each trigger interval, which introduces latency proportional to the directory size and does not provide sub-minute detection at scale. rescue mode captures unknown fields in a _rescued_data column but does not add them to the schema. WHY NOT B: Standard readStream with format('json') is essentially what Auto Loader uses internally, but without the file tracking checkpoint, notification integration, or schema inference capabilities. PERMISSIVE mode captures malformed records but does not handle schema evolution. WHY NOT D: backfillInterval is for catching up on missed files, not for real-time detection. Auto Loader fully supports JSON schema evolution natively — cloudFiles.schemaEvolutionMode is specifically designed for this. WHY NOT E: failOnNewColumns would cause the pipeline to fail every time a new JSON field is added, which is the opposite of graceful schema evolution and creates operational burden.