A data engineer wants an Auto Loader pipeline to run on a nightly schedule: each run should process every file accumulated since the previous run and then stop, keeping compute cost low rather than staying always-on. Which trigger meets this requirement?
Answer & explanation
Correct answer: B
WHY B: Trigger.AvailableNow processes all available files across multiple rate-limited micro-batches then stops, so scheduling it as a job lets compute terminate between runs — the cost-efficient batch pattern. WHY NOT A: Continuous stays always-on and does not pause when the directory drains, so compute keeps running. WHY NOT C: ProcessingTime is an always-on interval stream; the cluster stays provisioned, incurring continuous cost. WHY NOT D: Trigger.Once also stops, but it uses a single micro-batch with no rate limiting and is superseded by AvailableNow, so it is not required over it. WHY NOT E: maxFilesPerTrigger caps batch size but does not make a default stream stop; the query keeps running and waiting for new files.