Home / GenAI Engineer practice test / Application Development / Question

Databricks Generative AI Engineer Associate · free practice question

Following the deployment of a Generative AI assistant, the team establishes a continuous performance tracking system that assesses a daily sample of actual…

Exam-style question on Application Development from the free GenAI Engineer practice test, with the answer and a full explanation below.

Updated November 11, 2026

Application Development

Following the deployment of a Generative AI assistant, the team establishes a continuous performance tracking system that assesses a daily sample of actual user inquiries and system responses using the identical LLM-based quality indicators (e.g., factual accuracy, harmfulness). What key distinction exists between this ongoing monitoring activity and the initial pre-release offline assessment phase?

  1. AMonitoring applies the evaluation logic to live traffic continuously to catch degradation, while offline evaluation is a one-time gate before release.
  2. BMonitoring drops the LLM judges and instead tracks system-level statistics such as response latency and error rates across the sampled requests.
  3. CMonitoring relies on a different metric set than offline evaluation, because faithfulness can be measured in the pre-deployment stage but not on live traffic.
  4. DThere is no real distinction between the two, since Databricks recommends running one shared pipeline for both phases to avoid duplicated work.
Answer & explanation

Correct answer: A

WHY A is correct: Offline evaluation is a static, pre-deployment quality gate run on a fixed benchmark dataset to decide whether to release. Online monitoring is continuous and post-deployment, applying evaluation logic (including LLM judges) to live production samples to detect quality regression, data drift, and safety violations over time. The key differences are: data source (static benchmark vs. live traffic), frequency (point-in-time vs. continuous), and purpose (deployment gate vs. production health tracking).

WHY NOT C: Databricks explicitly supports using the same LLM judge configuration — including faithfulness — across both offline evaluation and online monitoring. Faithfulness measurement is not restricted to offline use.

WHY NOT B: Modern Gen AI monitoring systems (including Databricks Agent Evaluation) use LLM judges to assess quality dimensions well beyond system-level latency and error rates.

WHY NOT D: While the same LLM judge configuration is commonly reused across both phases, the processes differ fundamentally in dataset type, cadence, and purpose. Stating they are identical would eliminate the distinction that the Gen AI application lifecycle explicitly makes between them.

Take the full GenAI Engineer practice test free →

Would you pass today?

This question is one of 45 in the free, timed Databricks Generative AI Engineer Associate practice test. Take it and get a section-by-section score.

Start the free test →