Home / GenAI Engineer practice test / Evaluation and Monitoring

Free · 8 questions with explanations

Evaluation and Monitoring: Databricks Generative AI Engineer Associate Practice Questions

Exam-style questions on Evaluation and Monitoring. Pick your answer, then open the explanation to see why it's right — and why the other options are wrong.

Updated November 11, 2026

1 Evaluation and Monitoring

A team enables Databricks Agent Monitoring on a deployed LangChain agent. Agent Monitoring is configured with the same LLM judge metrics used in offline evaluation: faithfulness and answer relevance. What is the PRIMARY operational benefit of running Agent Monitoring continuously in production compared to running only periodic offline evaluations?

  1. AIt applies the judges to real production traffic, catching drift and regressions that a fixed offline dataset cannot represent
  2. BIt stores results in Unity Catalog with automatic versioning, whereas offline results live in MLflow runs without version history
  3. CIt removes the need for offline pre-deployment evaluation, since quality assurance is then covered after deployment
  4. DIt scores with a faster LLM judge model than offline evaluation, lowering the cost of each scoring call by roughly half
Show answer & explanation

Correct answer: A

WHY A is correct: The core advantage of continuous production monitoring over periodic offline evaluation is coverage of live, dynamic traffic. Production queries differ from curated evaluation sets — they include unanticipated phrasings, edge cases, domain shifts, and adversarial patterns. Agent Monitoring detects regressions caused by changes in user behavior, corpus drift, or downstream model updates in real time, allowing the team to respond before widespread user impact occurs. WHY NOT D: Agent Monitoring does not inherently use a 'faster' LLM judge model. Both offline and online evaluation can use the same judge, and cost differences depend on sampling rate, not a fixed 50% reduction. WHY NOT B: Both offline evaluation results (in MLflow) and Agent Monitoring results can be stored with versioning in Databricks. The storage mechanism is not the primary functional difference between them. WHY NOT C: Agent Monitoring and offline evaluation serve complementary roles in the Gen AI application lifecycle. Offline evaluation provides a controlled, reproducible quality gate before deployment. Production monitoring is not a replacement — the two phases are explicitly distinguished in the Gen AI lifecycle framework.

2 Evaluation and Monitoring

A company deploys multiple LLM-backed applications via Databricks AI Gateway. The security team needs to enforce that no single team can exceed 100,000 tokens per minute across all their calls, and the finance team needs a query-able log of all token consumption grouped by application for monthly cost allocation. Which AI Gateway features address BOTH requirements simultaneously?

  1. AModel signatures to cap each team at 100,000 tokens per minute, and AI Gateway usage tables to sum token use per application
  2. BRate limiting to cap each team at 100,000 tokens per minute, and MLflow experiment tags to sum tokens per application
  3. CRate limiting to cap each team at 100,000 tokens per minute, and AI Gateway usage tables to sum token use per application
  4. DModel signatures to cap each team at 100,000 tokens per minute, and MLflow experiment tags to sum token use per application
Show answer & explanation

Correct answer: C

WHY C: AI Gateway rate limiting configures per-route token-per-minute caps that are enforced at the gateway layer: requests exceeding the limit receive a 429 response without reaching the LLM, preventing budget overrun. AI Gateway usage tables capture cumulative token consumption per route/key/application in a structured, query-able format that the finance team can aggregate in SQL per application for monthly cost allocation. Both capabilities are native AI Gateway features; inference tables, by contrast, log endpoint-level request/response data and are not necessarily structured by application in the format needed for per-application cost allocation. WHY NOT B: Rate limiting covers the security requirement, but MLflow experiment tags are associated with training and evaluation runs, not with production AI Gateway serving requests. AI Gateway requests are not automatically tagged to MLflow experiments, so the tags cannot produce a token-consumption log for cost allocation. WHY NOT A: Usage tables cover the finance requirement, but model signatures define the input/output schema structure for validation purposes. They do not enforce rate limits, so no team would be capped at 100,000 tokens per minute. WHY NOT D: Neither half works: model signatures only describe input/output schemas and do not enforce rate limits, and MLflow experiment tags belong to training and evaluation runs rather than logging AI Gateway token consumption.

3 Evaluation and Monitoring

A company has deployed a customer-facing RAG chatbot for technical support. Six months post-launch, the team needs to establish ongoing production monitoring. Which set of metrics is MOST comprehensive for this specific deployment scenario?

  1. ATraining loss curve, validation perplexity from fine-tuning, GPU hours consumed per training epoch, and checkpoint size per run
  2. BLLM-judged faithfulness and answer relevance, p50/p95 latency, tokens per request, safety flag rate, and sampled Precision@K
  3. CBLEU score on a fixed benchmark, embedding model training set size, Delta table write throughput, and embedding dimension of the index
  4. DDaily active users, page load time of the chat UI, Vector Search index size on disk, and number of chat sessions opened per week
Show answer & explanation

Correct answer: B

WHY B is correct: Production RAG monitoring requires a multi-dimensional metric set: faithfulness and relevance catch quality degradation from document corpus drift; latency (p50/p95) detects serving infrastructure issues; token consumption drives cost monitoring; safety flag rate detects abuse pattern changes; and retrieval quality helps isolate whether degradation originates in the retriever vs the generator. WHY NOT A: Training loss and GPU hours are training-time metrics. They are irrelevant to a deployed, serving endpoint that has already been trained. Production monitoring tracks inference-time behavior, not training process health. WHY NOT D: UI page load time and index storage size are infrastructure concerns, not LLM quality or safety metrics. Daily active users is a product engagement metric, not a model quality signal for an AI chatbot. WHY NOT C: BLEU score requires ground-truth reference texts to compute — it cannot be calculated in real-time on live production traffic without pre-prepared references. Embedding model training set size and Delta write throughput are static configuration attributes, not ongoing monitoring signals.

4 Evaluation and Monitoring

A team uses the Databricks Agent Evaluation review app to collect SME ratings on their deployed agent's responses. Subject matter experts annotate responses with thumbs-up/thumbs-down ratings and optional free-text corrections. Which Databricks-native workflow BEST describes how to use these annotations to improve the agent in the next iteration?

  1. AExport the SME annotations from the review app to a CSV file and send it to the model vendor to request a custom fine-tuned model
  2. BAdd the thumbs-down annotations, with their corrections as expected responses, to the evaluation set and re-run mlflow.genai.evaluate()
  3. CDrop the free-text corrections and use the thumbs-up/thumbs-down ratings to compute a preference score per agent version in MLflow
  4. DUse the thumbs-down annotations as negative examples to train an RLHF reward model, and retire the offline evaluation dataset
Show answer & explanation

Correct answer: B

WHY B is correct: The Databricks review app integrates with Unity Catalog and persists SME annotations in a Delta table. The recommended workflow is: (1) stored annotations become new labeled evaluation examples (thumbs-down + free-text correction = question + expected_response pair), (2) these augment the offline evaluation dataset registered in MLflow Agent Evaluation, (3) re-running mlflow.genai.evaluate() with the augmented dataset against the updated agent measures whether the specific failure modes identified by SMEs have been resolved. Optionally, (4) high-quality curated annotation pairs can be used as supervised fine-tuning data for future model versions. WHY NOT A: Exporting to CSV and sharing with an external vendor is operationally slow, introduces data governance risks, and bypasses the Databricks-native annotation-to-evaluation pipeline that the review app is designed to support. WHY NOT D: RLHF (Reinforcement Learning from Human Feedback) is a valid but complex technique that requires significant infrastructure for reward model training. Using thumbs-down annotations exclusively for RLHF while retiring the offline evaluation dataset ignores simpler, faster improvement levers available in the Databricks platform, and without the evaluation dataset there is no way to measure whether the next iteration improved. WHY NOT C: Dropping the free-text corrections discards the most actionable signal in the SME annotations — the expert's statement of what the correct response should have been. Binary ratings alone provide a quality signal but not the expected response content needed to build evaluation examples or fine-tuning pairs.

5 Evaluation and Monitoring

A developer writes a custom scorer to evaluate whether an agent cites sources in its responses. The scorer should return 1.0 if at least one citation is present (e.g., '[Source: doc_name]') and 0.0 if no citation is found. Which implementation makes this check part of an MLflow evaluation run on Databricks, with a result recorded for every evaluated row?

  1. ADefine a plain function that returns 1.0 or 0.0, call it in a notebook loop over the responses and print the average citation rate
  2. BWrite a function that returns True or False for citation presence and record its result with mlflow.log_metric('citation_scorer', score) after each run
  3. CDecorate the check function with @scorer from mlflow.genai.scorers and pass it in the scorers list of mlflow.genai.evaluate()
  4. DBuild a LangChain Runnable that applies a regex tool to the response text and writes a binary citation label per row to a Delta table for later review
Show answer & explanation

Correct answer: C

WHY C is correct: In MLflow on Databricks a custom scorer is a Python function decorated with @scorer. It receives fields such as the outputs of each row and returns a number, a boolean or a Feedback object. Passing it in the scorers list of mlflow.genai.evaluate() runs it on every row, alongside any built-in scorers, and records the per-row results and the aggregate in the evaluation run. WHY NOT A: A loop in a notebook does compute the number, but outside the evaluation run. Nothing is attached to the evaluated rows or their traces, and the score does not appear next to the built-in judges in the evaluation results. WHY NOT B: mlflow.log_metric() records one number for a run. It does not produce a result per evaluated row, so the team cannot see which responses were missing citations. WHY NOT D: A chain that writes labels to a Delta table is a separate data pipeline. It is not part of the MLflow evaluation run, so its labels are not shown with the other evaluation results.

6 Evaluation and Monitoring

A team has enabled inference logging on a Databricks Model Serving endpoint for their RAG application. After one week of production traffic, they want to identify whether the retriever is returning relevant context for user queries. Which analysis using inference logs is MOST directly useful for this purpose?

  1. ASample query and retrieved-chunk pairs from the logs and score each pair for relevance with an LLM judge or manual review
  2. BCount requests per hour in the inference log table and plot the hourly traffic volume to find the periods of peak usage
  3. CCompare the average response length in tokens across time periods to detect whether answers are drifting toward more verbose output
  4. DCheck the inference log table for rows with NULL responses to identify failed requests and measure the endpoint's error rate
Show answer & explanation

Correct answer: A

WHY A is correct: Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production. Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production. WHY NOT B: Hourly traffic volume analysis is operational capacity planning, not retrieval quality assessment. It tells nothing about whether the retrieved chunks are relevant to user queries. WHY NOT C: Response verbosity over time may indicate model behavior drift but does not measure retrieval relevance. A brief response can still be grounded in highly relevant retrieved context, or a verbose response may be based on irrelevant chunks. WHY NOT D: Counting NULL responses measures endpoint reliability and error rate — an important operational metric but entirely separate from retrieval quality. NULL responses indicate system failures, not relevance degradation.

7 Evaluation and Monitoring

A data scientist runs mlflow.genai.evaluate() on a RAG agent with built-in LLM judges as scorers. The evaluation dataset contains 50 question-answer pairs. After the run, which information is available from the evaluation results that CANNOT be obtained from simple batch inference alone?

  1. AThe raw response text that the agent generated for each of the 50 questions
  2. BPer-example LLM judge scores and aggregated quality metrics for the run
  3. CThe Python package versions recorded in the run's environment metadata
  4. DThe total wall-clock time the agent took to generate the 50 responses
Show answer & explanation

Correct answer: B

WHY B: mlflow.genai.evaluate() runs the scorers (LLM judges such as correctness, groundedness and relevance) against each example in the evaluation dataset and attaches each judge's feedback to that example's trace. It aggregates the scores into run-level metrics and logs them as an evaluation run in the MLflow experiment, enabling comparison against previous evaluation runs and app versions in the same experiment. None of this — judge scoring, aggregation, and run comparison — is produced by simple batch inference, which only generates raw model outputs. WHY NOT A: Raw model response text is produced by both batch inference and mlflow.genai.evaluate(). It is the input the judges score, not a result unique to the evaluation function. WHY NOT D: Wall-clock inference time can be measured by any timed batch inference loop. MLflow logs latency metrics more precisely, but wall-clock time measurement alone is not unique to mlflow.genai.evaluate(). WHY NOT C: Package names and versions in the evaluation environment are logged as part of MLflow's run environment metadata — a side effect of any MLflow run, not a unique feature of mlflow.genai.evaluate().

8 Evaluation and Monitoring

A team is setting up automated evaluation for a deployed knowledge base assistant. They have an evaluation dataset of 200 questions but NO labeled correct answers — only the user questions and retrieved source documents. Which judges can they apply to this dataset WITHOUT needing to first add ground-truth labels? Select ALL that apply.

  1. AAnswer correctness, which compares the generated answer with a reference answer to verify factual accuracy
  2. BFaithfulness, which checks whether each claim in the response is supported by the retrieved source documents
  3. CAnswer relevance, which checks whether the generated response actually addresses the user's question
  4. DRetrieval precision, which checks whether the retrieved document chunks are relevant to the user's query
  5. EDocument recall, which measures what fraction of the correct reference documents were retrieved
Show answer & explanation

Correct answer: BCD

WHY B is correct: Faithfulness needs only the response and the retrieved context it was generated from. Both are available without any ground-truth labels, so this judge can run on the dataset as it is. WHY C is correct: Answer relevance needs only the user's question and the generated response, both of which exist at inference time, so no labeled answer is required. WHY D is correct: Retrieval precision judges whether each retrieved chunk is relevant to the query. It needs only the query and the retrieved chunks, not a labeled answer or a labeled document set. WHY NOT A: Answer correctness compares the generated answer against a labeled reference answer. Without expected responses in the dataset the judge has nothing to compare against, so it cannot produce a score. WHY NOT E: Document recall measures what fraction of the relevant documents were retrieved, so you must already know which documents are the right ones for each query. That labeled set is a form of ground truth the team does not have.

Take the full GenAI Engineer practice test →