A team is evaluating their RAG application, which is deployed with inference logging on a Databricks Model Serving endpoint. After a week of production use, they want to determine if the retriever component is successfully providing relevant contextual information for incoming user queries. Which of the following analytical approaches, utilizing the inference logs, would be most appropriate for this assessment?
Answer & explanation
Correct answer: B
WHY B is correct: Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production. Inference logs capture the full request payload, including retrieved context chunks when logged as part of the input. Sampling logged query–chunk pairs and scoring retrieval relevance directly measures the retriever's production performance on real user traffic — the exact goal stated. This identifies whether the retriever is failing on specific query types or corpus sections in production.
WHY NOT A: Hourly traffic volume analysis is operational capacity planning, not retrieval quality assessment. It tells nothing about whether the retrieved chunks are relevant to user queries.
WHY NOT C: Response verbosity over time may indicate model behavior drift but does not measure retrieval relevance. A brief response can still be grounded in highly relevant retrieved context, or a verbose response may be based on irrelevant chunks.
WHY NOT D: Counting NULL responses measures endpoint reliability and error rate — an important operational metric but entirely separate from retrieval quality. NULL responses indicate system failures, not relevance degradation.