Skip to main content

2 posts tagged with "online evaluation"

View All Tags

AI Agent Evaluation vs Observability: What's the Difference?

· 18 min read

AI agent observability records what an agent did: every model call, tool call, and intermediate step, with latency, token counts, and errors. AI agent evaluation scores whether what it did was good, using code checks, LLM judges, or human review. Observability gives you the evidence. Evaluation gives you the verdict. You need both, and the traces you collect for the first are the raw material for the second.

The failure this guide is about is a quiet one: the dashboard stays green, every status code is 200, and the agent still gives a wrong answer. Observability alone cannot catch that, and the two words are often used as if they meant the same thing, so we wrote down where the line sits. It is for engineers who run an agent in production, or are about to.

Disclosure: this guide was written by MLflow maintainers. The examples use MLflow Tracing and mlflow.genai.evaluate, and the concepts hold in any tool.

TL;DR​

Six points to take away.

  • Observability answers "what happened?" It captures traces of each run: inputs, outputs, tool calls, latency, cost, and errors.
  • Evaluation answers "was it good?" It applies graders (scorers) to an agent's output and its trajectory, and returns scores you can compare across versions.
  • They share one data model. A trace is the input to both a dashboard and a scorer, so tracing is the first step for either.
  • Observability is reactive; offline evaluation is proactive. Anthropic's engineering team says production monitoring detects problems after users hit them, while automated evals run before launch and on every commit.
  • Online evaluation is the bridge. It scores a sample of live traces with the same judges you use in CI, so production monitoring reports quality, not only errors.
  • A healthy agent can still be wrong. Zero errors and fast responses say nothing about whether the answer was correct.

Table of Contents​

What is AI agent observability?​

AI agent observability is the practice of recording enough about every agent run that you can reconstruct and inspect it afterwards. For an agent, that means a trace: a structured record of the request, each model call, each tool call with its arguments and result, and the final response.

MLflow's tracing documentation describes the same idea: tracing captures "the inputs, outputs, and metadata associated with each intermediate step of a request," and it records latency and token usage at each step. The same page describes MLflow Tracing as fully OpenTelemetry-compatible, which matters if your traces need to flow into tools you already run.

Observability is built for questions like these:

  • Why did this request take 40 seconds?
  • Which tool call failed, and with what arguments?
  • How many tokens did this user session cost?
  • Which step did the agent take just before it looped?

Notice what these questions have in common. Each one is about the run, not about whether the result was right. A trace can show that an agent called issue_refund with a valid order ID and got a 200 back. It cannot tell you that the customer was not eligible for a refund.

Quick answer: Agent observability tells you what an agent did on a given run: the steps, tool calls, latency, cost, and errors. It does not, by itself, tell you whether the outcome was correct.

What is AI agent evaluation?​

AI agent evaluation is the practice of measuring whether an agent does its job well, using repeatable tests that return scores. Anthropic's engineering team defines an eval as "a test for an AI system: give an AI an input, then apply grading logic to its output to measure success" in its post Demystifying evals for AI agents (January 9, 2026).

Three terms from that post are worth borrowing:

  • Grader: "logic that scores some aspect of the agent's performance." A single task can have several graders.
  • Transcript (also called a trace or trajectory): "the complete record of a trial, including outputs, tool calls, reasoning," and intermediate results.
  • Outcome: the final state of the environment. A booking agent's outcome is whether the reservation exists in the database, not whether the agent says it does.

The same post says agent evaluations typically combine three grader types: code-based, model-based, and human. Code-based graders are fast, cheap, and objective but brittle. Model-based graders handle nuance but are non-deterministic and need calibration against human judgment. Human graders are the most reliable, but they are expensive and slow, so they are mostly used to calibrate the model-based ones.

Evaluation is built for questions like these:

  • Did the new prompt make answers more correct than the old one?
  • Does the agent pick the right tool for this request?
  • Did the change we shipped on Tuesday cause a regression?
  • Is this LLM judge's score trustworthy?

Quick answer: Agent evaluation applies graders to an agent's outputs and trajectories to produce scores. It tells you whether a run, or a version of the agent, is good enough, and lets you compare versions.

How do evaluation and observability differ?​

The cleanest split is by the question each one answers and the moment it answers it.

DimensionObservabilityEvaluation
Core questionWhat happened on this run?Was the result good, and is this version better?
Main artifactTraces, spans, metrics, logsScores, pass rates, regression reports
Who judgesA person reading a trace, or alert rules on metricsScorers: code checks, LLM judges, human labels
When it runsContinuously on live trafficOffline on a dataset (CI, before release) and online on samples
Typical failure foundErrors, timeouts, loops, cost spikesWrong answers, wrong tools, policy violations, regressions
Needs ground truth?NoSometimes (expected answers); judges can run without it
Output is aRecord you inspectNumber you compare

Quick answer: Observability is a record of behavior that you inspect. Evaluation is a measurement of quality that you compare. Observability answers "what happened?" and evaluation answers "was it good?"

A second difference is timing. Anthropic's post makes the point directly: production monitoring tracks "metrics and errors in live systems," and it is reactive, because problems reach users before you detect them. Evals run without real users and can run on every commit. In the post's words, automated evals are "especially useful pre-launch and in CI/CD," while monitoring "kicks in post-launch to detect distribution drift and unanticipated real-world failures." It concludes that the most effective teams combine the two.

The third difference is what a failure looks like. Observability surfaces loud failures: exceptions, timeouts, a tool returning an error. Agents also fail quietly. They answer fluently and wrongly, call a tool that was allowed but not appropriate, or skip a required step. A dashboard of green status codes will not show any of that. Scorers will, because they read the content and the trajectory rather than the status.

Where do the two overlap?​

The overlap is a single idea: the trace. Once an agent emits traces, three things become possible that were not possible before.

Scorers can read the trace. MLflow's documentation on evaluating traces says "Scorers have access to the complete MLflow traces, including spans, attributes, and outputs." That lets you grade the path as well as the answer, for example the tool call trajectory, sub-agent routing, or retrieved document recall.

Production traces become test cases. Langfuse's documentation describes evaluation as running both offline, "before you ship a change," and online, on live production traces. The point is not which tool you use. The point is that a failure you see in a production trace should end up as a case in your offline suite.

Evaluation scores become observability data. When a judge scores a sampled share of live traffic, quality becomes a time series next to latency and cost. You can alert on "correctness dropped from 0.91 to 0.78" the same way you alert on a latency spike.

                 ┌──────────────────────────────┐
│ Agent in production │
└──────────────┬───────────────┘
│ emits
▼
┌─────────────────┐
│ Traces │ ← observability
└───┬─────────┬───┘
inspect, alert │ │ sample + score
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ Dashboards │ │ Online judges │ ← evaluation
└──────────────┘ └────────┬─────────┘
│ failures become test cases
▼
┌──────────────────┐
│ Offline dataset │ → CI gate
└──────────────────┘

Do you need both for an AI agent?​

Yes, for any agent that does real work. Each one covers a blind spot in the other.

Observability without evaluation leaves you able to explain every failure you notice and unable to notice the quiet ones. You find out about wrong answers from users, which is the slowest and most expensive channel. You also have no way to say whether Tuesday's prompt change helped.

Evaluation without observability gives you scores with no explanation. A correctness score of 0.72 is a number. The trace behind a failing case is the reason. Without traces you cannot see which step went wrong, and your dataset will be built from imagined inputs rather than real ones.

A rule of thumb that holds up in practice:

If you are...Start with
Prototyping, no users yetTracing on, plus a small offline eval set
Shipping to a first group of real usersTracing, error and latency alerts, sampled online judges
Changing prompts, models, or tools oftenAn offline eval suite as a CI gate
Running a multi-step or multi-agent systemTrace-reading scorers for tool calls and routing

Quick answer: Use observability to see and debug what your agent does. Use evaluation to measure whether it does it well. Run offline evaluation before each release, and online evaluation on a sample of live traces after.

A worked example: one trace, two jobs​

Take a support agent that can look up orders and issue refunds. A customer writes, "My order arrived broken, I want my money back."

The agent looks up the order, decides the order qualifies, calls issue_refund, and replies "Done, your refund is on its way."

What observability shows. The trace has four spans: the user request, a model call that chose the lookup_order tool, the tool call itself, and a second model call that chose issue_refund. Total latency is 3.2 seconds, the cost is a few thousand tokens, and every status code is 200. Nothing is red.

What evaluation shows. A scorer that checks policy finds that the order was placed 94 days ago and the refund window is 30 days. The agent should have escalated to a human. A second scorer, which checks whether the response promised something the policy does not allow, fails too.

Neither tool alone tells the full story. The dashboard says the system is healthy. The score says the behavior is wrong. The trace then lets you see exactly where: the model never compared the order date with the policy. That is the loop in practice. Evaluation finds the problem, observability explains it, and the failing trace becomes a permanent test case.

(This scenario is an illustration we constructed, not data from a customer.)

How to run both with MLflow​

MLflow is an open-source AI engineering platform governed by the Linux Foundation, and it uses one trace for both jobs. The steps below use APIs from the MLflow GenAI documentation. Treat the code as a starting point and check parameter names against the docs for your installed version.

Step 1: Trace the agent​

Add the decorator to the functions that make up a run, and mark tools as tool spans.

import mlflow

mlflow.set_experiment("support-agent")


@mlflow.trace(span_type="TOOL")
def lookup_order(order_id: str) -> dict:
...


@mlflow.trace(span_type="TOOL")
def issue_refund(order_id: str) -> dict:
...


@mlflow.trace(span_type="AGENT")
def support_agent(question: str) -> str:
...

For supported libraries, mlflow.<library>.autolog() traces calls with one line, for example mlflow.openai.autolog(). For production, the tracing docs recommend the lightweight mlflow-tracing package, which they describe as a 95% smaller footprint than the full mlflow package, and with async logging enabled, trace logging runs in the background so it does not slow the application. For more on instrumenting, see our pages on agent observability and LLM tracing.

Step 2: Evaluate offline against a dataset​

An evaluation has three parts: a dataset, scorers, and a predict function. The dataset holds inputs and optional expectations.

from mlflow.genai.scorers import Correctness, Guidelines

dataset = [
{
"inputs": {"question": "My order from 94 days ago arrived broken. Refund me."},
"expectations": {"expected_response": "Escalates to a human; no refund issued."},
},
]

results = mlflow.genai.evaluate(
data=dataset,
predict_fn=support_agent,
scorers=[
Correctness(),
Guidelines(
name="refund_policy",
guidelines="Never promise a refund for an order older than 30 days.",
),
],
)

This is the same pattern we describe in Continuous LLM Evaluation, including how to build the dataset and wire the run into CI.

Step 3: Score the trajectory, not only the answer​

Built-in scorers include ToolCallCorrectness, which judges whether the agent called the right tools with the right arguments, ToolCallEfficiency, which flags redundant tool calls, and ConversationCompleteness and UserFrustration for multi-turn conversations. You can also write a code scorer that reads the trace directly.

from mlflow.entities import SpanType, Trace
from mlflow.genai.scorers import ToolCallCorrectness, scorer


@scorer
def at_most_one_refund(trace: Trace) -> bool:
refunds = trace.search_spans(span_type=SpanType.TOOL, name="issue_refund")
return len(refunds) <= 1


results = mlflow.genai.evaluate(
data=dataset,
predict_fn=support_agent,
scorers=[ToolCallCorrectness(), at_most_one_refund],
)

Code scorers are free and exact, so use them for hard rules like "never refund twice". Use LLM judges for qualities code cannot express. If you rely on judges, calibrate them first; our guide to LLM judge calibration shows how.

Step 4: Evaluate traces you already have​

You do not need to re-run the agent to score it. Traces from mlflow.search_traces() can be passed to mlflow.genai.evaluate() directly, with no predict_fn. The documentation notes the function "will automatically extract the inputs, outputs, and other intermediate information from the trace object."

from datetime import datetime, timedelta

since = int((datetime.now() - timedelta(days=1)).timestamp() * 1000)
traces = mlflow.search_traces(filter_string=f"timestamp > {since}")

results = mlflow.genai.evaluate(data=traces, scorers=[ToolCallCorrectness()])

This is how you turn yesterday's production traffic into a quality report, and how you reuse one set of traces across several evaluation runs without paying to regenerate them.

Step 5: Score live traffic automatically​

MLflow's automatic evaluation runs LLM judges on traces and multi-turn conversations as they are logged. It runs asynchronously, so it does not slow the application. You register a judge that uses a gateway model, then start it with a sample rate. Replace my-llm-endpoint with the name of your own AI Gateway endpoint.

from mlflow.genai.scorers import Guidelines, ScorerSamplingConfig

refund_judge = Guidelines(
name="refund_policy",
guidelines="Never promise a refund for an order older than 30 days.",
model="gateway:/my-llm-endpoint",
).register()

refund_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=0.2))

The documentation lists prerequisites: a running MLflow server, tracing enabled, and a configured AI Gateway endpoint for the judge model. Multi-turn evaluation needs session IDs on the traces. It suggests a high sample rate during development and a lower one in production to control cost. The docs also note that failed evaluations are not retried automatically, so treat a missing score as "not evaluated", not "passed".

With all five steps in place, one experiment holds traces you can inspect, offline runs you can compare, and live quality scores you can chart.

Common mistakes​

Treating a green dashboard as a quality signal. Latency, error rate, and token counts describe health. They say nothing about correctness. Add at least one scorer that reads the content.

Evaluating only the final answer. An agent can reach the right answer through a wrong or wasteful path, such as calling a destructive tool when a read-only one would do. Score the trajectory too.

Building the eval set from imagined inputs. Real traces show what users actually ask. Seed your dataset from them and add every production failure you find.

Trusting an uncalibrated judge. A model-based grader is a second black box. Compare its scores to human labels on a sample before you gate a release on it.

Using the same sample for tuning and reporting. If you iterate on prompts until the offline suite passes, the suite stops measuring generalization. Keep a held-out set and refresh it from production.

Scoring every live trace forever. Judge calls cost money and add load to your gateway. Start with a high sample rate, then lower it once the signal is stable.

Buying two tools when one data model would do. Many platforms in this space, including Langfuse, LangSmith, Arize Phoenix, Braintrust, and MLflow, offer both tracing and evaluation. The useful question is whether the same trace feeds both, so you do not copy data between systems. Our comparison of agent observability tools and agent evaluation frameworks cover how each one handles this.

Is "AI observability" the same as "LLM observability"?​

Not quite, and the difference matters for the words you search with. LLM observability usually means watching individual model calls: prompts, completions, latency, tokens, and cost. AI observability, or agent observability, covers the larger system around those calls: tool use, retrieval, multi-step plans, multiple agents, and sessions that span many turns. A single agent run contains many LLM calls, and the failures that matter often sit between them.

In practice the tooling is the same. If your traces capture the model calls, the tool calls, and how they nest, you have both. We cover the model-call layer in more depth in our guide to open-source LLM tracing tools.

Where to start​

If you have no tracing today, turn it on first. Everything else depends on it, and it costs one decorator or one autolog() line. If you have tracing and no evaluation, pick the three worst failures you have seen in production, turn them into dataset rows, and write one scorer for each. That is a real eval suite, and you can grow it from there.

Then run it. The MLflow GenAI documentation has quickstarts for tracing and evaluation, and the MLflow GitHub repository is the place to report issues or ask questions.

FAQ​

What is the difference between AI agent evaluation and observability?​

Observability records what an agent did on a run: its steps, tool calls, latency, cost, and errors. Evaluation scores whether that behavior was good, using code checks, LLM judges, or human review. Observability is something you inspect, and evaluation is something you compare across runs and versions.

Is agent evaluation part of observability?​

They are separate practices that share data. Some platforms bundle them, and some treat online evaluation as part of observability because it scores live traces. A useful test: if it tells you what happened, it is observability. If it returns a quality score you can compare, it is evaluation.

What is the difference between offline and online evaluation?​

Offline evaluation runs your agent against a fixed dataset before you ship a change, typically in CI. Online evaluation scores a sample of live production traces after release. Offline catches regressions early, and online catches problems your dataset did not anticipate.

Can I do agent evaluation without tracing?​

You can score final outputs without traces, but you lose the ability to grade tool calls, routing, and other intermediate steps, and you lose the explanation for a failing score. For agents, tracing is the better foundation.

Do I need an LLM judge to evaluate an agent?​

No. Code-based checks cover hard rules like valid output format, required tool calls, and latency budgets, and they are cheap and exact. LLM judges are for qualities code cannot express, such as helpfulness or tone, and should be calibrated against human labels.

Which comes first, observability or evaluation?​

Observability, in most cases. Tracing is quick to add, and the traces it collects supply the inputs for your first evaluation dataset. Add scorers as soon as you have traces from real or realistic usage.

Does MLflow support both agent evaluation and observability?​

Yes. Open-source MLflow provides OpenTelemetry-compatible tracing and mlflow.genai.evaluate with built-in and custom scorers. The same traces can be inspected in the UI, passed to evaluation, and scored automatically as they arrive. MLflow is Apache-2.0 licensed and hosted by the Linux Foundation. Any managed MLflow offering from a vendor is a separate product from the open-source project.

Next step​

Add @mlflow.trace to one agent function today, run a handful of real requests, and open the MLflow UI to read the traces. Then pick one failure you found and write the first scorer for it with the MLflow evaluation guide.

How to Evaluate an AI Agent Using Production Traces

· 15 min read

To evaluate an AI agent using production traces, record every run as a trace, pull a sample of those traces, and score them without re-running the agent. Use trace-based judges for the path the agent took (which tools it called, with what arguments) and session-level judges for whole conversations. Then turn each failure into a test case, so the next version has to pass it.

Last reviewed: November 14, 2026.

This guide is for engineers who already have an agent in production and a tracing setup, and want to know whether the agent is any good. We work on MLflow, so the code uses MLflow. The steps carry over to any tool that stores full agent traces.

What you will build​

  • A tagged trace stream, with tool calls and conversation sessions recorded
  • Trace-level scores for tool selection, tool arguments, and efficiency
  • Session-level scores for completeness and user frustration
  • A custom judge for one failure mode that matters to your product
  • A loop that sends failed production traces back into a dataset
  • Automatic scoring of a sample of live traffic

If you need the broader program around this, such as CI gates, golden datasets, and a 30-day rollout, read Continuous LLM Evaluation first. This guide does not repeat it. It goes deeper on one question: how do you judge an agent from what it did in production?

Why can't you evaluate an agent on its final answer alone?​

An agent can return a correct answer after a wrong path: a tool called with the wrong arguments, a retry loop that burned tokens, or a lookup that returned nothing and was ignored. Scoring the answer alone misses all of it, and the path is where agents break.

Reliability is the second problem. In the τ-bench paper, researchers tested tool-using agents in simulated customer-service domains. They report that the function-calling agents of the time, including gpt-4o, succeed on fewer than 50% of tasks, and are inconsistent: pass^8, the chance of solving the same task correctly in all of 8 attempts, is below 25% in the retail domain.

One passing run says little. You need many real runs, scored the same way, and production is the only place you get them in volume. The paper is from 2024 and tested gpt-4o-class models, so newer models will score higher, but the lesson about repeated runs has not changed.

What is trace-based agent evaluation?​

Trace-based agent evaluation scores an agent from the recorded trace of its run, instead of re-running it on a test input. A trace holds the user input, every model call, every tool call with its arguments and result, and the final output, so a judge can grade the whole path.

This has three practical benefits:

  • It scores real behavior. Production users ask things you did not predict.
  • It is cheap to repeat. You generate traces once and re-score them with new judges, without paying for the agent's model calls again. The MLflow docs on evaluating traces note this reuse for offline evaluation.
  • It sees the intermediate steps. Scorers get the full trace, including spans and attributes, so they can check the tool-call trajectory and not only the output.

Anthropic's engineering team makes the same split in Demystifying evals for AI agents. Offline evals run on thousands of tasks without touching users, but "can create false confidence if it doesn't match real usage patterns." Production monitoring "reveals real user behavior at scale" but is reactive. Their advice is to combine automated evals for iteration, production monitoring for ground truth, and periodic human review for calibration.

Which tool should you use for production-trace evaluation?​

MLflow is a good fit when you want tracing, judges, datasets, and automatic scoring in one open-source server. It is Apache-2.0 licensed and a Linux Foundation project.

Disclosure: this guide was written by MLflow maintainers. Read the MLflow-specific code as one option. Other platforms such as Langfuse, LangSmith, and Braintrust take their own approach to scoring production traces, and our comparison of agent observability tools covers the trade-offs. The workflow below does not depend on the tool.

StepWhat you scoreMLflow API
RecordNothing yet; capture tools and sessions@mlflow.trace(span_type="TOOL"), mlflow.update_current_trace
Score the pathTool choice, arguments, redundant callsToolCallCorrectness, ToolCallEfficiency
Score the answerRelevance to the requestRelevanceToQuery
Score the dialogueCompleteness, frustration across turnsConversationCompleteness, UserFrustration
Score a custom caseA failure only your product hasmake_judge with {{ trace }}
Keep scoringA sample of live trafficScorer.register() and .start()

Step 1: How do you make traces that an evaluator can use?​

Mark tool functions with span_type="TOOL" and tag each trace with a session ID. The built-in tool-call judges need at least one span of type TOOL, and the conversation judges need a session ID to group turns.

import mlflow

mlflow.set_experiment("support-agent")


@mlflow.trace(span_type="TOOL")
def lookup_order(order_id: str) -> dict:
return {"order_id": order_id, "status": "shipped"}


@mlflow.trace(span_type="AGENT")
def support_agent(question: str, session_id: str) -> str:
mlflow.update_current_trace(metadata={"mlflow.trace.session": session_id})
order = lookup_order("A-1042")
return f"Your order {order['order_id']} has {order['status']}."

If you use a framework such as LangGraph or the OpenAI SDK, call its autolog() function instead of decorating by hand, and still set the session ID. For the full list of what a trace captures, see our page on LLM and agent observability.

Put the session ID on every turn. Without it, a five-turn conversation looks like five unrelated requests, and conversation-level scores are impossible.

Step 2: How do you pull production traces to evaluate?​

Search the tracking server for a recent window and ask for trace objects. A recent window means you score the agent you actually shipped.

from datetime import datetime, timedelta

since = datetime.now() - timedelta(days=1)
traces = mlflow.search_traces(
filter_string=f"timestamp > {int(since.timestamp() * 1000)}",
max_results=200,
return_type="list",
)

The trace search guide lists the supported filters, such as status and tags. Do not sample only successes. Errors, slow runs, and runs where a user gave a thumbs-down are the cases you most need to see, so include them on purpose.

Step 3: How do you score tool calls from a trace?​

Use the two built-in tool-call judges. ToolCallCorrectness checks whether the agent picked the right tools and passed correct arguments for the request. ToolCallEfficiency checks for redundant calls. Neither needs ground-truth labels in its default mode.

from mlflow.genai.scorers import (
RelevanceToQuery,
ToolCallCorrectness,
ToolCallEfficiency,
)

results = mlflow.genai.evaluate(
data=traces,
scorers=[
ToolCallCorrectness(),
ToolCallEfficiency(),
RelevanceToQuery(),
],
)

You do not pass a predict_fn. MLflow reads the inputs, outputs, and spans from each trace. The judges are LLM-based, and the default judge model is an OpenAI model, so set OPENAI_API_KEY or pass model= to use another provider (on a Databricks tracking server the default differs). See the tool-call judge docs for modes that compare against expected tool calls.

Open the run in the MLflow UI and read the rationale on the lowest-scoring traces. Anthropic's guide says plainly that "you won't know if your graders are working well unless you read the transcripts and grades from many trials."

Step 4: How do you evaluate multi-turn conversations?​

Group traces into sessions and use the session-level judges. ConversationCompleteness returns whether the assistant addressed all of the user's requests by the end. UserFrustration returns none, resolved, or unresolved, so you can tell a rough patch that recovered from one that ended badly.

from mlflow.genai.scorers import ConversationCompleteness, UserFrustration

sessions = mlflow.search_sessions(max_results=50)
session_traces = [trace for session in sessions for trace in session]

results = mlflow.genai.evaluate(
data=session_traces,
scorers=[ConversationCompleteness(), UserFrustration()],
)

Multi-turn evaluation needs MLflow 3.10 or later, and the docs mark the feature as experimental, so expect the API to change. Our post on multi-turn evaluation walks through the sessions view and a simulation workflow for conversations you have not seen yet.

Session scores catch things that no single turn shows: an agent that forgets a constraint from turn two, or answers each question well while the user's real goal never gets met.

Step 5: How do you write a judge for a failure only your agent has?​

Write a trace-based judge with make_judge. When the instructions contain {{ trace }}, the judge gets the full trace and can inspect spans on its own. Name one specific failure, give clear categories, and keep it to one question per judge.

from typing import Literal

from mlflow.genai.judges import make_judge

refund_judge = make_judge(
name="refund_policy_followed",
instructions=(
"Analyze the execution {{ trace }}. Decide whether the agent checked "
"the order status with a tool before promising a refund.\n"
"Answer 'checked' if a lookup tool ran before the refund promise, "
"'skipped' if the agent promised a refund without a lookup, and "
"'not_applicable' if no refund was discussed."
),
feedback_value_type=Literal["checked", "skipped", "not_applicable"],
model="openai:/gpt-5-mini",
)

results = mlflow.genai.evaluate(data=traces, scorers=[refund_judge])

Set a model explicitly on a trace-based judge, as in the example. The custom judge docs show more examples, including loop detection and tool optimization.

Do not trust a new judge on day one. The MT-Bench paper, Judging LLM-as-a-Judge, found in 2023 that strong judges like GPT-4 reach over 80% agreement with human preferences on MT-Bench and Chatbot Arena, the same level humans agree with each other. The same abstract lists position, verbosity, and self-enhancement biases. Airbnb's engineering team recommends a golden dataset of 50 to 100 examples that includes bad examples for calibrating a judge, and the same mix works for a regression set. Label a set that size by hand, compare, and rewrite the instructions until the judge agrees with you. Our guide to LLM judge calibration covers the method, and the LLM-as-a-judge overview explains the scorer types.

Step 6: How do you turn failed production traces into regression tests?​

Annotate the failing traces with the expected behavior, then add them to an evaluation dataset. The next version of the agent runs against that dataset, so a fixed failure stays fixed.

from mlflow.genai.datasets import create_dataset

failing_ids = ["<failing-trace-id>"]
for trace_id in failing_ids:
mlflow.log_expectation(
trace_id=trace_id,
name="expected_tool_calls",
value=[{"name": "lookup_order", "arguments": {"order_id": "A-1042"}}],
)

dataset = create_dataset(
name="support_agent_regressions",
experiment_id=mlflow.get_experiment_by_name("support-agent").experiment_id,
)
dataset.merge_records([mlflow.get_trace(trace_id) for trace_id in failing_ids])

Choose the traces you merge. A dataset full of easy cases hides regressions, so add the failures plus a few clean cases, which proves your scorers can tell good from bad.

To run the dataset against a new agent version, call mlflow.genai.evaluate(data=dataset, predict_fn=support_agent, scorers=[...]). The dataset workflow guide shows the full path.

This is the loop that one paper proposes under the name evaluation-driven development: the paper on evaluation-driven development and operations of LLM agents argues that evaluation should run through development and operations as a continuous function, not a gate before launch.

Step 7: How do you score live traffic automatically?​

Register a judge and start it with a sampling rate. MLflow then scores new traces and sessions as they arrive, with no scheduled job to maintain. This is online evaluation, and it is how you notice drift without waiting for a user complaint.

from mlflow.genai.scorers import ScorerSamplingConfig, ToolCallCorrectness

judge = ToolCallCorrectness(model="gateway:/my-llm-endpoint")
registered = judge.register(name="tool_call_correctness")
registered.start(sampling_config=ScorerSamplingConfig(sample_rate=0.2))

Three limits to know from the automatic evaluation docs:

  • It supports LLM judges, built-in or from make_judge. Plain code scorers are not supported on a standard server.
  • The judge model runs through an AI Gateway endpoint, which you configure first.
  • A new judge scores traces and sessions that are at most one hour old. It does not backfill history, so use mlflow.genai.evaluate for that.

Start at a low sample rate. Judge calls cost money, and a small slice of traffic is enough to see a trend. Raise the rate for a new release, then lower it again.

What mistakes should you avoid?​

Most failed trace-evaluation programs fail in the same few ways.

  • Scoring only the answer. You miss wrong tool calls that happened to produce a plausible reply.
  • Skipping the human check. The Who Validates the Validators study describes "criteria drift": people need criteria to grade outputs, but grading outputs helps them define the criteria. Expect your rubric to change after you read real traces.
  • One judge for everything. A single "is this good?" judge gives a vague score. Use one judge per failure mode.
  • Mixing up prompts and sampling. If you change both a judge prompt and the sample at once, you cannot tell whether quality moved or the measurement did. Change one at a time.
  • Sampling only what users flagged. Complaints are a biased sample. Score a random slice too.
  • Treating one run as a result. Given the pass^k results above, repeat the important cases and look at the spread.

When is this approach the wrong choice?​

Trace-based evaluation needs real traffic, so it does not help before launch. For a new agent, build a small dataset by hand or simulate conversations, then switch to production traces once users arrive. It also needs privacy review: traces contain user input, so check what you store and who can read it before you point judges at it.

It does not replace outcome checks either. If you can verify the final state directly, such as a refund row in a database, write a code check and use the LLM judge only for what code cannot see. Anthropic's guidance is to choose deterministic graders where possible and LLM graders where necessary.

FAQ​

How do I evaluate an AI agent using production traces?​

Capture each run as a trace, pull a sample, and score it with judges that read the whole trace. In MLflow that means mlflow.search_traces to fetch the sample and mlflow.genai.evaluate with scorers such as ToolCallCorrectness and UserFrustration. Then add the failures to a dataset to guard against regressions.

How is trace-based evaluation different from offline evaluation?​

Offline evaluation runs the agent on a fixed dataset you chose in advance. Trace-based evaluation scores runs that already happened, so it reflects real user behavior but cannot test a new version by itself. Use both: traces to find failures, a dataset to test the fix.

How many production traces do I need?​

There is no fixed number. Start with the code above, which pulls up to 200 traces from the last day. Include errors and slow runs on purpose, and read the lowest scores by hand. Once your judges agree with your labels, add automatic scoring at a low sample rate.

Can I evaluate multi-turn conversations, not just single requests?​

Yes. Set a session ID on every trace, then use session-level judges such as ConversationCompleteness and UserFrustration. This needs MLflow 3.10 or later and is marked experimental in the docs.

Do I need ground-truth labels?​

Not to start. The default tool-call judges and the conversation judges work without labels. Labels help later: add expected tool calls to failing traces so your regression tests can check exact behavior.

What does it cost to run LLM judges on production traffic?​

Each judged trace is at least one extra model call, and a trace-based judge that explores a long trace costs more. Control it with the sample rate, a cheaper judge model for simple checks, and a filter string that targets only the traces you care about.

Next step​

Pick one agent and one failure mode this week. Add span_type="TOOL" to its tools, pull a day of recent traces, and run ToolCallCorrectness over them. Read the ten lowest scores, then follow the MLflow guide to evaluating production traces to build your first judge. For the monitoring side, see MLflow AI monitoring, and for the wider evaluation toolkit, see LLM and agent evaluation in MLflow.

Sources​

Every external claim in this guide comes from one of these sources.