A team runs a nightly scheduled Lakeflow Job in production. It is currently attached to a shared all-purpose cluster that several analysts also use interactively during the day. The team wants to lower cost and improve isolation for the scheduled run. Which change best meets both goals?
Answer & explanation
Correct answer: D
WHY D: A job cluster is provisioned for the run and terminated when it finishes, so it is cheaper than an always-on all-purpose cluster and isolates the scheduled run from interactive users. WHY NOT A: An all-purpose cluster is billed at interactive rates and stays configured for shared use; it is not the cost-optimal choice for a scheduled job even when scaled down. WHY NOT B: A serverless SQL warehouse runs SQL only and cannot execute Python or notebook tasks, so it cannot host this job. WHY NOT C: Raising concurrency shares the cluster more, not less; it does not isolate the job and does not reduce cost. WHY NOT E: A continuous trigger keeps compute running constantly, which increases cost rather than reducing it for a once-nightly job.