A team evaluating an AI agent for legal document analysis observes that the platform's standard LLM evaluation metrics (e.g., for faithfulness, relevance, toxicity) do not adequately assess a crucial domain-specific quality: whether the agent fully identifies all mandatory legal provisions pertinent to their jurisdiction (termed 'clause completeness'). What is the appropriate method to integrate this specific evaluation criterion within Databricks Agent Evaluation?
Answer & explanation
Correct answer: C
WHY C is correct: Databricks Agent Evaluation's custom Scorers API allows teams to define domain-specific quality dimensions as Python functions decorated with @scorer. The scorer receives the agent's input and response, applies custom logic (clause presence checks via regex or an LLM call against a list of mandatory clauses), and returns a pass/fail value, a number or a Feedback object. This custom scorer is passed in the scorers list of mlflow.genai.evaluate() alongside the built-in judges, integrating clause completeness into the standard evaluation pipeline.
WHY NOT A: Databricks custom Scorers are designed precisely so teams don't need to wait for Databricks to add new judges. The self-service custom scorer API addresses domain-specific needs without platform modifications.
WHY NOT B: Repurposing the relevance score with prompt reinterpretation is semantically incorrect — relevance measures topical alignment to the query, not completeness of specific required content. This would produce unreliable, misleading scores for clause completeness.
WHY NOT D: Databricks Agent Evaluation explicitly supports custom Scorers as a first-class feature for extending evaluation beyond built-in judges. The claim that only pre-built judges are supported is factually incorrect.