Agent Evaluation in MLflow with Google ADK

MLflow's evaluation framework now wraps the Google Agent Development Kit (ADK) evaluator suite as third-party scorers. The integration covers five ADK evaluators (ToolTrajectory, ResponseMatch, ResponseEvaluation, Safety, Hallucination) so agent evaluations can use ADK's built-in metrics inside any mlflow.genai.evaluate() run.

Agent evaluation isn't shaped like RAG evaluation. A wrong tool call early in a trajectory can produce a correct-looking final response. A fluent answer can still skip a required step. ADK's evaluator suite was built to answer three questions about the full trace: did the agent call the right tools in the right order, did the final response match what the user needed, and did the model hallucinate. The provided scorers include both deterministic (e.g., tool trajectory matching, response matching based on ROUGE) and LLM-judge checks (e.g., safety and hallucination scores). All five run against the same MLflow trace, so a single evaluation can layer cheap structural checks under more expensive judge calls.

Available Scorers

The five Google ADK scorers, grouped by how they score:

Scorer	What it evaluates	How it works
ToolTrajectory	Whether the agent called the right tools in the right order	Wraps ADK's TrajectoryEvaluator, comparing actual vs expected tool-call sequence
ResponseMatch	Lexical similarity between actual and expected response	Wraps ADK's RougeEvaluator and returns the ROUGE-1 F-measure
ResponseEvaluation	Whether the final response matches expected semantically	Wraps ADK's FinalResponseMatchV2Evaluator, an LLM judge with majority voting
Safety	Whether the response contains unsafe content	Wraps ADK's SafetyEvaluatorV1, which delegates to Vertex AI's prebuilt SAFETY metric
Hallucination	Whether the response contains hallucinated content	Wraps ADK's HallucinationsV1Evaluator, an LLM judge

The deterministic scorers (ToolTrajectory, ResponseMatch) read actual tool calls and the final response directly from the MLflow trace. The LLM-judge scorers (ResponseEvaluation, Hallucination) call gemini-2.5-flash by default with five-sample majority voting, and both arguments are configurable. Safety is the exception. It routes through Vertex AI's managed SAFETY metric, which ignores judge-model overrides, so the scorer raises if model or num_samples is passed.

Evaluating an Agent

The example below runs trajectory matching, ROUGE response similarity, and an LLM-judge response evaluation against a flight-booking agent, scoring recorded interactions without re-running the agent.

import mlflow
from mlflow.genai.scorers.google_adk import (
    ToolTrajectory,
    ResponseMatch,
    ResponseEvaluation,
)

# Each row carries what the agent actually did (outputs, actual_tool_calls)
# alongside the ground truth (expected_response, expected_tool_calls). For an
# online setup where the agent runs inside the eval, pass a predict_fn and
# the scorers will read tool calls from the MLflow trace directly.
eval_data = [
    {
        "inputs": {"query": "Find me a flight to Paris next Friday."},
        "outputs": "I found 3 flights to Paris on Friday: AA101, DL202, UA303.",
        "expectations": {
            "expected_tool_calls": [
                {"name": "search_flights", "args": {"destination": "Paris"}},
            ],
            "actual_tool_calls": [
                {"name": "search_flights", "args": {"destination": "Paris"}},
            ],
            "expected_response": "Here are flights to Paris next Friday.",
        },
    },
    {
        "inputs": {"query": "What is the cheapest flight to Tokyo?"},
        "outputs": "The cheapest flight to Tokyo is JL789 at $850.",
        "expectations": {
            "expected_tool_calls": [
                {"name": "search_flights", "args": {"destination": "Tokyo"}},
                {"name": "sort_by_price", "args": {}},
            ],
            "actual_tool_calls": [
                {"name": "search_flights", "args": {"destination": "Tokyo"}},
                {"name": "sort_by_price", "args": {}},
            ],
            "expected_response": "The cheapest flight to Tokyo is JL789.",
        },
    },
]

results = mlflow.genai.evaluate(
    data=eval_data,
    scorers=[
        ToolTrajectory(match_type="EXACT", threshold=0.5),
        ResponseMatch(threshold=0.5),
        ResponseEvaluation(threshold=0.6),
        # Safety and Hallucination slot into this same list when needed.
        # Safety also needs Google Cloud credentials (Vertex AI) on top of
        # GEMINI_API_KEY because it delegates to a managed Vertex AI metric.
    ],
)

The second row exercises a multi-tool trajectory. The agent has to call search_flights first and sort_by_price second. ToolTrajectory scores 1.0 when the call sequence matches under the selected match_type (EXACT, IN_ORDER, or ANY_ORDER) and 0.0 when it doesn't. ResponseMatch returns a ROUGE-1 F-measure independent of tool calls. ResponseEvaluation asks the judge whether the final response matches semantically. Results land in the same MLflow evaluation UI as any other scorer, so trajectory failures, lexical mismatches, and judge disagreements are visible side-by-side.

How It Works

The integration follows the same scorer pattern as MLflow's other third-party integrations: the scorer maps MLflow inputs, outputs, expectations, and trace into the invocation shape ADK expects, calls the underlying evaluator, and converts the result into an MLflow Feedback object.

The deterministic path extracts the actual tool calls from the MLflow trace by walking the trace's TOOL spans, so the scorer sees what the agent actually did, not what a test fixture says it did. That's how the integration avoids a class of silent false negatives where a test passes against a stuffed string but the production extractor reads a different shape. The LLM-judge path forwards the inputs and outputs to ADK's invocation format and lets the judge model (default gemini-2.5-flash) score against the prompt template baked into the underlying evaluator.

One detail worth flagging: model names for the LLM-judge scorers must be resolvable by ADK's LlmRegistry (gemini-2.5-flash, gemini-2.5-pro, etc.). MLflow model URIs like databricks or openai:/gpt-4o aren't supported here, because ADK's evaluators wire directly into Google's model registry. If you need a different judge backend, the TruLens and DeepEval integrations cover the broader provider set.

Getting Started

pip install "mlflow>=3.13" "google-adk"

For the LLM-judge scorers, set GEMINI_API_KEY. For Safety, also set GOOGLE_CLOUD_PROJECT and GOOGLE_CLOUD_LOCATION and authenticate with gcloud auth application-default login or a service account, because Vertex AI's managed SAFETY metric runs on Google Cloud project compute. See the MLflow Google ADK scorer documentation for the full API reference and configuration options.

Resources

• MLflow Google ADK scorer documentation
• MLflow Evaluation documentation
• MLflow Tracing quickstart
• Google ADK Evaluation guide
• Vertex AI Gen AI Evaluation Service

Provenance

I contributed the Google ADK scorer integration across two MLflow PRs: #22299 for the deterministic scorers and #22496 for the LLM judges, both merged by B-Step62. Trace-based trajectory extraction landed in response to review feedback, and real-API verification on Gemini covered each scoring outcome before merge.