Skip to main content

2 posts tagged with "agent observability"

View All Tags

AI Agent Evaluation vs Observability: What's the Difference?

· 18 min read

AI agent observability records what an agent did: every model call, tool call, and intermediate step, with latency, token counts, and errors. AI agent evaluation scores whether what it did was good, using code checks, LLM judges, or human review. Observability gives you the evidence. Evaluation gives you the verdict. You need both, and the traces you collect for the first are the raw material for the second.

The failure this guide is about is a quiet one: the dashboard stays green, every status code is 200, and the agent still gives a wrong answer. Observability alone cannot catch that, and the two words are often used as if they meant the same thing, so we wrote down where the line sits. It is for engineers who run an agent in production, or are about to.

Disclosure: this guide was written by MLflow maintainers. The examples use MLflow Tracing and mlflow.genai.evaluate, and the concepts hold in any tool.

TL;DR​

Six points to take away.

  • Observability answers "what happened?" It captures traces of each run: inputs, outputs, tool calls, latency, cost, and errors.
  • Evaluation answers "was it good?" It applies graders (scorers) to an agent's output and its trajectory, and returns scores you can compare across versions.
  • They share one data model. A trace is the input to both a dashboard and a scorer, so tracing is the first step for either.
  • Observability is reactive; offline evaluation is proactive. Anthropic's engineering team says production monitoring detects problems after users hit them, while automated evals run before launch and on every commit.
  • Online evaluation is the bridge. It scores a sample of live traces with the same judges you use in CI, so production monitoring reports quality, not only errors.
  • A healthy agent can still be wrong. Zero errors and fast responses say nothing about whether the answer was correct.

Table of Contents​

What is AI agent observability?​

AI agent observability is the practice of recording enough about every agent run that you can reconstruct and inspect it afterwards. For an agent, that means a trace: a structured record of the request, each model call, each tool call with its arguments and result, and the final response.

MLflow's tracing documentation describes the same idea: tracing captures "the inputs, outputs, and metadata associated with each intermediate step of a request," and it records latency and token usage at each step. The same page describes MLflow Tracing as fully OpenTelemetry-compatible, which matters if your traces need to flow into tools you already run.

Observability is built for questions like these:

  • Why did this request take 40 seconds?
  • Which tool call failed, and with what arguments?
  • How many tokens did this user session cost?
  • Which step did the agent take just before it looped?

Notice what these questions have in common. Each one is about the run, not about whether the result was right. A trace can show that an agent called issue_refund with a valid order ID and got a 200 back. It cannot tell you that the customer was not eligible for a refund.

Quick answer: Agent observability tells you what an agent did on a given run: the steps, tool calls, latency, cost, and errors. It does not, by itself, tell you whether the outcome was correct.

What is AI agent evaluation?​

AI agent evaluation is the practice of measuring whether an agent does its job well, using repeatable tests that return scores. Anthropic's engineering team defines an eval as "a test for an AI system: give an AI an input, then apply grading logic to its output to measure success" in its post Demystifying evals for AI agents (January 9, 2026).

Three terms from that post are worth borrowing:

  • Grader: "logic that scores some aspect of the agent's performance." A single task can have several graders.
  • Transcript (also called a trace or trajectory): "the complete record of a trial, including outputs, tool calls, reasoning," and intermediate results.
  • Outcome: the final state of the environment. A booking agent's outcome is whether the reservation exists in the database, not whether the agent says it does.

The same post says agent evaluations typically combine three grader types: code-based, model-based, and human. Code-based graders are fast, cheap, and objective but brittle. Model-based graders handle nuance but are non-deterministic and need calibration against human judgment. Human graders are the most reliable, but they are expensive and slow, so they are mostly used to calibrate the model-based ones.

Evaluation is built for questions like these:

  • Did the new prompt make answers more correct than the old one?
  • Does the agent pick the right tool for this request?
  • Did the change we shipped on Tuesday cause a regression?
  • Is this LLM judge's score trustworthy?

Quick answer: Agent evaluation applies graders to an agent's outputs and trajectories to produce scores. It tells you whether a run, or a version of the agent, is good enough, and lets you compare versions.

How do evaluation and observability differ?​

The cleanest split is by the question each one answers and the moment it answers it.

DimensionObservabilityEvaluation
Core questionWhat happened on this run?Was the result good, and is this version better?
Main artifactTraces, spans, metrics, logsScores, pass rates, regression reports
Who judgesA person reading a trace, or alert rules on metricsScorers: code checks, LLM judges, human labels
When it runsContinuously on live trafficOffline on a dataset (CI, before release) and online on samples
Typical failure foundErrors, timeouts, loops, cost spikesWrong answers, wrong tools, policy violations, regressions
Needs ground truth?NoSometimes (expected answers); judges can run without it
Output is aRecord you inspectNumber you compare

Quick answer: Observability is a record of behavior that you inspect. Evaluation is a measurement of quality that you compare. Observability answers "what happened?" and evaluation answers "was it good?"

A second difference is timing. Anthropic's post makes the point directly: production monitoring tracks "metrics and errors in live systems," and it is reactive, because problems reach users before you detect them. Evals run without real users and can run on every commit. In the post's words, automated evals are "especially useful pre-launch and in CI/CD," while monitoring "kicks in post-launch to detect distribution drift and unanticipated real-world failures." It concludes that the most effective teams combine the two.

The third difference is what a failure looks like. Observability surfaces loud failures: exceptions, timeouts, a tool returning an error. Agents also fail quietly. They answer fluently and wrongly, call a tool that was allowed but not appropriate, or skip a required step. A dashboard of green status codes will not show any of that. Scorers will, because they read the content and the trajectory rather than the status.

Where do the two overlap?​

The overlap is a single idea: the trace. Once an agent emits traces, three things become possible that were not possible before.

Scorers can read the trace. MLflow's documentation on evaluating traces says "Scorers have access to the complete MLflow traces, including spans, attributes, and outputs." That lets you grade the path as well as the answer, for example the tool call trajectory, sub-agent routing, or retrieved document recall.

Production traces become test cases. Langfuse's documentation describes evaluation as running both offline, "before you ship a change," and online, on live production traces. The point is not which tool you use. The point is that a failure you see in a production trace should end up as a case in your offline suite.

Evaluation scores become observability data. When a judge scores a sampled share of live traffic, quality becomes a time series next to latency and cost. You can alert on "correctness dropped from 0.91 to 0.78" the same way you alert on a latency spike.

                 ┌──────────────────────────────┐
│ Agent in production │
└──────────────┬───────────────┘
│ emits
▼
┌─────────────────┐
│ Traces │ ← observability
└───┬─────────┬───┘
inspect, alert │ │ sample + score
▼ ▼
┌──────────────┐ ┌──────────────────┐
│ Dashboards │ │ Online judges │ ← evaluation
└──────────────┘ └────────┬─────────┘
│ failures become test cases
▼
┌──────────────────┐
│ Offline dataset │ → CI gate
└──────────────────┘

Do you need both for an AI agent?​

Yes, for any agent that does real work. Each one covers a blind spot in the other.

Observability without evaluation leaves you able to explain every failure you notice and unable to notice the quiet ones. You find out about wrong answers from users, which is the slowest and most expensive channel. You also have no way to say whether Tuesday's prompt change helped.

Evaluation without observability gives you scores with no explanation. A correctness score of 0.72 is a number. The trace behind a failing case is the reason. Without traces you cannot see which step went wrong, and your dataset will be built from imagined inputs rather than real ones.

A rule of thumb that holds up in practice:

If you are...Start with
Prototyping, no users yetTracing on, plus a small offline eval set
Shipping to a first group of real usersTracing, error and latency alerts, sampled online judges
Changing prompts, models, or tools oftenAn offline eval suite as a CI gate
Running a multi-step or multi-agent systemTrace-reading scorers for tool calls and routing

Quick answer: Use observability to see and debug what your agent does. Use evaluation to measure whether it does it well. Run offline evaluation before each release, and online evaluation on a sample of live traces after.

A worked example: one trace, two jobs​

Take a support agent that can look up orders and issue refunds. A customer writes, "My order arrived broken, I want my money back."

The agent looks up the order, decides the order qualifies, calls issue_refund, and replies "Done, your refund is on its way."

What observability shows. The trace has four spans: the user request, a model call that chose the lookup_order tool, the tool call itself, and a second model call that chose issue_refund. Total latency is 3.2 seconds, the cost is a few thousand tokens, and every status code is 200. Nothing is red.

What evaluation shows. A scorer that checks policy finds that the order was placed 94 days ago and the refund window is 30 days. The agent should have escalated to a human. A second scorer, which checks whether the response promised something the policy does not allow, fails too.

Neither tool alone tells the full story. The dashboard says the system is healthy. The score says the behavior is wrong. The trace then lets you see exactly where: the model never compared the order date with the policy. That is the loop in practice. Evaluation finds the problem, observability explains it, and the failing trace becomes a permanent test case.

(This scenario is an illustration we constructed, not data from a customer.)

How to run both with MLflow​

MLflow is an open-source AI engineering platform governed by the Linux Foundation, and it uses one trace for both jobs. The steps below use APIs from the MLflow GenAI documentation. Treat the code as a starting point and check parameter names against the docs for your installed version.

Step 1: Trace the agent​

Add the decorator to the functions that make up a run, and mark tools as tool spans.

import mlflow

mlflow.set_experiment("support-agent")


@mlflow.trace(span_type="TOOL")
def lookup_order(order_id: str) -> dict:
...


@mlflow.trace(span_type="TOOL")
def issue_refund(order_id: str) -> dict:
...


@mlflow.trace(span_type="AGENT")
def support_agent(question: str) -> str:
...

For supported libraries, mlflow.<library>.autolog() traces calls with one line, for example mlflow.openai.autolog(). For production, the tracing docs recommend the lightweight mlflow-tracing package, which they describe as a 95% smaller footprint than the full mlflow package, and with async logging enabled, trace logging runs in the background so it does not slow the application. For more on instrumenting, see our pages on agent observability and LLM tracing.

Step 2: Evaluate offline against a dataset​

An evaluation has three parts: a dataset, scorers, and a predict function. The dataset holds inputs and optional expectations.

from mlflow.genai.scorers import Correctness, Guidelines

dataset = [
{
"inputs": {"question": "My order from 94 days ago arrived broken. Refund me."},
"expectations": {"expected_response": "Escalates to a human; no refund issued."},
},
]

results = mlflow.genai.evaluate(
data=dataset,
predict_fn=support_agent,
scorers=[
Correctness(),
Guidelines(
name="refund_policy",
guidelines="Never promise a refund for an order older than 30 days.",
),
],
)

This is the same pattern we describe in Continuous LLM Evaluation, including how to build the dataset and wire the run into CI.

Step 3: Score the trajectory, not only the answer​

Built-in scorers include ToolCallCorrectness, which judges whether the agent called the right tools with the right arguments, ToolCallEfficiency, which flags redundant tool calls, and ConversationCompleteness and UserFrustration for multi-turn conversations. You can also write a code scorer that reads the trace directly.

from mlflow.entities import SpanType, Trace
from mlflow.genai.scorers import ToolCallCorrectness, scorer


@scorer
def at_most_one_refund(trace: Trace) -> bool:
refunds = trace.search_spans(span_type=SpanType.TOOL, name="issue_refund")
return len(refunds) <= 1


results = mlflow.genai.evaluate(
data=dataset,
predict_fn=support_agent,
scorers=[ToolCallCorrectness(), at_most_one_refund],
)

Code scorers are free and exact, so use them for hard rules like "never refund twice". Use LLM judges for qualities code cannot express. If you rely on judges, calibrate them first; our guide to LLM judge calibration shows how.

Step 4: Evaluate traces you already have​

You do not need to re-run the agent to score it. Traces from mlflow.search_traces() can be passed to mlflow.genai.evaluate() directly, with no predict_fn. The documentation notes the function "will automatically extract the inputs, outputs, and other intermediate information from the trace object."

from datetime import datetime, timedelta

since = int((datetime.now() - timedelta(days=1)).timestamp() * 1000)
traces = mlflow.search_traces(filter_string=f"timestamp > {since}")

results = mlflow.genai.evaluate(data=traces, scorers=[ToolCallCorrectness()])

This is how you turn yesterday's production traffic into a quality report, and how you reuse one set of traces across several evaluation runs without paying to regenerate them.

Step 5: Score live traffic automatically​

MLflow's automatic evaluation runs LLM judges on traces and multi-turn conversations as they are logged. It runs asynchronously, so it does not slow the application. You register a judge that uses a gateway model, then start it with a sample rate. Replace my-llm-endpoint with the name of your own AI Gateway endpoint.

from mlflow.genai.scorers import Guidelines, ScorerSamplingConfig

refund_judge = Guidelines(
name="refund_policy",
guidelines="Never promise a refund for an order older than 30 days.",
model="gateway:/my-llm-endpoint",
).register()

refund_judge.start(sampling_config=ScorerSamplingConfig(sample_rate=0.2))

The documentation lists prerequisites: a running MLflow server, tracing enabled, and a configured AI Gateway endpoint for the judge model. Multi-turn evaluation needs session IDs on the traces. It suggests a high sample rate during development and a lower one in production to control cost. The docs also note that failed evaluations are not retried automatically, so treat a missing score as "not evaluated", not "passed".

With all five steps in place, one experiment holds traces you can inspect, offline runs you can compare, and live quality scores you can chart.

Common mistakes​

Treating a green dashboard as a quality signal. Latency, error rate, and token counts describe health. They say nothing about correctness. Add at least one scorer that reads the content.

Evaluating only the final answer. An agent can reach the right answer through a wrong or wasteful path, such as calling a destructive tool when a read-only one would do. Score the trajectory too.

Building the eval set from imagined inputs. Real traces show what users actually ask. Seed your dataset from them and add every production failure you find.

Trusting an uncalibrated judge. A model-based grader is a second black box. Compare its scores to human labels on a sample before you gate a release on it.

Using the same sample for tuning and reporting. If you iterate on prompts until the offline suite passes, the suite stops measuring generalization. Keep a held-out set and refresh it from production.

Scoring every live trace forever. Judge calls cost money and add load to your gateway. Start with a high sample rate, then lower it once the signal is stable.

Buying two tools when one data model would do. Many platforms in this space, including Langfuse, LangSmith, Arize Phoenix, Braintrust, and MLflow, offer both tracing and evaluation. The useful question is whether the same trace feeds both, so you do not copy data between systems. Our comparison of agent observability tools and agent evaluation frameworks cover how each one handles this.

Is "AI observability" the same as "LLM observability"?​

Not quite, and the difference matters for the words you search with. LLM observability usually means watching individual model calls: prompts, completions, latency, tokens, and cost. AI observability, or agent observability, covers the larger system around those calls: tool use, retrieval, multi-step plans, multiple agents, and sessions that span many turns. A single agent run contains many LLM calls, and the failures that matter often sit between them.

In practice the tooling is the same. If your traces capture the model calls, the tool calls, and how they nest, you have both. We cover the model-call layer in more depth in our guide to open-source LLM tracing tools.

Where to start​

If you have no tracing today, turn it on first. Everything else depends on it, and it costs one decorator or one autolog() line. If you have tracing and no evaluation, pick the three worst failures you have seen in production, turn them into dataset rows, and write one scorer for each. That is a real eval suite, and you can grow it from there.

Then run it. The MLflow GenAI documentation has quickstarts for tracing and evaluation, and the MLflow GitHub repository is the place to report issues or ask questions.

FAQ​

What is the difference between AI agent evaluation and observability?​

Observability records what an agent did on a run: its steps, tool calls, latency, cost, and errors. Evaluation scores whether that behavior was good, using code checks, LLM judges, or human review. Observability is something you inspect, and evaluation is something you compare across runs and versions.

Is agent evaluation part of observability?​

They are separate practices that share data. Some platforms bundle them, and some treat online evaluation as part of observability because it scores live traces. A useful test: if it tells you what happened, it is observability. If it returns a quality score you can compare, it is evaluation.

What is the difference between offline and online evaluation?​

Offline evaluation runs your agent against a fixed dataset before you ship a change, typically in CI. Online evaluation scores a sample of live production traces after release. Offline catches regressions early, and online catches problems your dataset did not anticipate.

Can I do agent evaluation without tracing?​

You can score final outputs without traces, but you lose the ability to grade tool calls, routing, and other intermediate steps, and you lose the explanation for a failing score. For agents, tracing is the better foundation.

Do I need an LLM judge to evaluate an agent?​

No. Code-based checks cover hard rules like valid output format, required tool calls, and latency budgets, and they are cheap and exact. LLM judges are for qualities code cannot express, such as helpfulness or tone, and should be calibrated against human labels.

Which comes first, observability or evaluation?​

Observability, in most cases. Tracing is quick to add, and the traces it collects supply the inputs for your first evaluation dataset. Add scorers as soon as you have traces from real or realistic usage.

Does MLflow support both agent evaluation and observability?​

Yes. Open-source MLflow provides OpenTelemetry-compatible tracing and mlflow.genai.evaluate with built-in and custom scorers. The same traces can be inspected in the UI, passed to evaluation, and scored automatically as they arrive. MLflow is Apache-2.0 licensed and hosted by the Linux Foundation. Any managed MLflow offering from a vendor is a separate product from the open-source project.

Next step​

Add @mlflow.trace to one agent function today, run a handful of real requests, and open the MLflow UI to read the traces. Then pick one failure you found and write the first scorer for it with the MLflow evaluation guide.

How to Trace AI Agents and MCP Tool Calls End to End

· 16 min read

To trace an AI agent end to end, record every step of one run as a span inside a single trace: the user request, each LLM call, each tool or MCP call, and the final answer. Then group the traces by session. With MLflow Tracing, that takes a tracking server, one autolog call for your framework, and one decorator around each MCP tool call.

This guide is for engineers whose agent works in a demo and does something odd in production, and who need to see what the agent actually did. We build MLflow, so the examples use MLflow Tracing. The steps apply to any OpenTelemetry-compatible tracer.

Disclosure: the MLflow team wrote this guide.

TL;DR​

An agent trace is a tree of spans, one per step, and you add the MCP step yourself.

  • An agent run is a tree: one root AGENT span, with LLM and TOOL spans below it. A trace is that whole tree.
  • Frameworks such as the OpenAI Agents SDK, LangGraph, and the Claude Agent SDK can be traced with one autolog() call. Hand-written agent loops need the @mlflow.trace decorator.
  • An MCP tool call is a JSON-RPC tools/call request. Plain MCP client code does not record it as a span, so wrap the call in a TOOL span (or check that your framework does it for you, as PydanticAI does).
  • Use a session_id to join the traces of a multi-turn conversation. Use W3C trace-context headers over HTTP to join traces across services.
  • Everything here is OpenTelemetry-compatible, so you can send the same traces to another backend.

What does "end to end" mean for an AI agent?​

End to end means that one trace shows the full path from the user's request to the final answer, including every model call and every tool call in between. If a step is missing from the trace, you cannot say why the agent chose a wrong tool or returned a wrong answer.

A normal web request is mostly deterministic. An agent is not. The model decides at run time which tool to call, with which arguments, and how many times. Two runs of the same prompt can take different paths. That is why logs of the final answer do not help much. You need the path.

MLflow's span model is built for this. A span is one step. Spans nest in a tree, and the tree is the trace. MLflow defines span types such as AGENT (an autonomous agent operation), LLM, RETRIEVER, and TOOL (a tool execution, typically by an agent). The span object is compatible with the OpenTelemetry span spec and is exported as standard OTLP.

Here is the shape of one traced run in this guide (an illustrative example):

trace: "Where is order A-1001?"
└── run_agent (AGENT) whole run
├── chat completion (LLM) model picks get_order_status
├── call_mcp_tool (TOOL) tools/call get_order_status
└── chat completion (LLM) model writes the answer

The LLM spans in this diagram come from autologging a provider SDK. The AGENT and TOOL spans are the two you add yourself for a hand-written loop. The code below produces the AGENT and TOOL pair; we ran it and checked the span tree (see Step 3).

Who this guide is for​

This guide is for you if you build an agent that calls tools and want to answer three questions from a trace: which tool did the agent call, what did it return, and where did the time go.

It works for framework agents and hand-written loops. It also covers MCP servers, because the Model Context Protocol (MCP) is an open standard for connecting agents to tools. If your agent has no tools, a plain LLM trace is enough, and the guide to open-source LLM tracing tools is a better start.

Why MLflow for agent and MCP tracing​

MLflow is a good fit for this job if you want tracing, evaluation, and storage in one Apache-2.0 server, and you want your traces in a vendor-neutral format. MLflow is backed by the Linux Foundation. MLflow Tracing is OpenTelemetry-native: the tracking server accepts traces over OTLP at /v1/traces from any language, and traces made with the MLflow SDK can be exported to other OpenTelemetry backends.

That is not the same as "best for every setup". If you only need instrumentation and already run an observability backend, a lighter library may be enough. If your agent is not written in Python or TypeScript, you can still send traces over OTLP, but you lose the one-line autologging. The MLflow tracing overview lists what is supported.

How do you trace an AI agent end to end? Six steps​

Follow these steps in order. Each step adds one kind of visibility and works on its own.

Step 1: Run a tracking server​

The server stores traces and shows them in the trace UI. For local work, start one with:

pip install mlflow
mlflow server --port 5000

In your agent code, point MLflow at it and name an experiment. Set the experiment before you turn on tracing, so every trace lands in the same place:

import mlflow

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my-agent")

For a quick test without a server, a local SQLite URI such as sqlite:///mlflow.db also works. We ran the example in Step 3 against a SQLite URI; swap it in for the server URL there to run without a server.

Step 2: Turn on autologging for your agent framework​

If you use a supported framework, one call traces it. Each framework has its own autolog function:

FrameworkCallNote
OpenAI Agents SDKmlflow.openai.autolog()Captures agent handoffs and the calls to the OpenAI API (docs)
LangGraphmlflow.langchain.autolog()LangGraph tracing extends the LangChain integration (docs)
Claude Agent SDK (Python)mlflow.anthropic.autolog()Needs MLflow 3.5 or newer. Traces ClaudeSDKClient; a direct call to query() is not traced (docs)
PydanticAImlflow.pydantic_ai.autolog()Also captures MCP call_tool and list_tools (docs)

Call the autolog function once, before you create the agent or client. After that, each agent run produces a trace with its LLM calls and tool calls.

PydanticAI deserves a note. Its integration records MCP server interactions, including call_tool and list_tools, as individual spans in the trace UI. If you use PydanticAI with MCP, you may not need Step 3 at all. Check one trace first.

The full list of integrations is in the MLflow tracing docs. If your framework is not listed, go to Step 3.

Step 3: Trace MCP tool calls​

An MCP client calls a tool by sending a tools/call request to the server, as the MCP specification describes. The spec defines two standard transports, stdio and Streamable HTTP, and the call looks the same to your code on both. If your client library or framework does not trace that call, the trace shows a gap: the model asked for a tool, and then the answer appears, with nothing in between.

Close the gap by wrapping the call in a TOOL span. Wrap the whole run in an AGENT span. This example uses the official MCP Python SDK and a tiny local server.

First, the server:

server.py
from mcp.server.fastmcp import FastMCP

mcp = FastMCP("orders")


@mcp.tool()
def get_order_status(order_id: str) -> str:
"""Look up the status of an order."""
return f"Order {order_id} shipped"


if __name__ == "__main__":
mcp.run()

Then the traced client:

client.py
import asyncio
import sys
from pathlib import Path

import mlflow
from mcp import ClientSession, StdioServerParameters
from mcp.client.stdio import stdio_client
from mlflow.entities import SpanType

mlflow.set_tracking_uri("http://localhost:5000")
mlflow.set_experiment("my-agent")


@mlflow.trace(span_type=SpanType.TOOL)
async def call_mcp_tool(session: ClientSession, name: str, arguments: dict):
result = await session.call_tool(name, arguments)
return result.content[0].text


@mlflow.trace(span_type=SpanType.AGENT)
async def run_agent(question: str) -> str:
params = StdioServerParameters(command=sys.executable, args=[str(Path(__file__).with_name("server.py"))])
async with stdio_client(params) as (read, write):
async with ClientSession(read, write) as session:
await session.initialize()
# A real agent lets the LLM choose the tool. It is hard-coded here.
return await call_mcp_tool(
session, "get_order_status", {"order_id": "A-1001"}
)


print(asyncio.run(run_agent("Where is order A-1001?")))

We ran this with MLflow 3.18.0 and mcp 1.30.0. The output was Order A-1001 shipped, and the trace held a root run_agent span of type AGENT with a child call_mcp_tool span of type TOOL. Because @mlflow.trace records a function's inputs and outputs, the tool span shows the tool name, the arguments, and the text the server returned.

Two notes on this code:

  • Pin mcp<2 if you copy it. The mcp package moved to a 2.x line in which FastMCP was renamed MCPServer, so this server code needs 1.x.
  • In a real agent, add your LLM provider's autolog call (Step 2). The LLM spans then appear as siblings of the tool span inside the same AGENT span, and you can read the whole decision loop in order.

Name the span after the tool, not after your wrapper, when you have many tools. A trace full of call_mcp_tool rows is hard to scan. The @mlflow.trace(name=...) argument sets the name, and mlflow.start_span(name=..., span_type=SpanType.TOOL) does the same when you need a span inside a block instead of a function.

Step 4: Group traces into sessions​

A chat agent produces one trace per turn. To debug a conversation, you need the turns together. MLflow lets you attach a session ID and a user ID to every trace created in a block of code:

import mlflow

mlflow.openai.autolog()

with mlflow.tracing.context(session_id="session-123", user="user-456"):
agent_turn("What is the status of order A-1001?") # your agent's entry point

The session_id and user parameters need MLflow 3.12.0 or newer. On older versions, set the mlflow.trace.session or mlflow.trace.user key in the trace metadata. See Track Users & Sessions.

In a web app, wrap the request handler with this context manager. Every trace from that request then carries the user and session without any change to your agent code.

Step 5: Connect traces across services​

Often the MCP server, or a sub-agent, runs in a different process or on a different host. Without extra work, each process makes its own trace, and you must match them by hand.

MLflow's distributed tracing fixes this for HTTP calls. It passes the active trace context in headers that follow the W3C TraceContext standard. The caller, inside an active span, uses get_tracing_context_headers_for_http_request to get the headers. The callee uses set_tracing_context_from_http_request_headers to read them, and its spans join the same trace. Both services must log to the same tracking server and the same experiment, or the spans end up in separate traces. The distributed tracing guide has a runnable two-service example.

There is a limit you should know about. That guide shows HTTP services. A stdio MCP server has no HTTP headers, so this method does not apply to it. We have not tested header propagation through the MCP Streamable HTTP transport, so we do not claim it works. For a stdio server, trace the tool function inside the server with @mlflow.trace, log to the same experiment, and tag both traces with the same session ID so you can find them together.

Step 6: Keep your traces portable with OpenTelemetry​

MLflow ingests and exports traces in the OpenTelemetry format, including the OpenTelemetry GenAI semantic conventions. Two things follow:

  • An agent written in Java, Go, or Rust can send spans to the MLflow server. Set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to <tracking-uri>/v1/traces, and set OTEL_EXPORTER_OTLP_TRACES_HEADERS to x-mlflow-experiment-id=<experiment-id> so the server knows which experiment gets the spans.
  • Traces from the MLflow SDK can go to other OpenTelemetry backends, so you are not locked in.

Read the OpenTelemetry integration page for setup, and the OpenTelemetry tracing announcement for background.

What should you look for in an agent trace?​

Open a trace and read the tree from top to bottom. These five patterns cover the agent failures we see most often in traces:

Pattern in the traceWhat it usually means
A TOOL span with an error, followed by a retry loopThe tool failed and the agent keeps trying. Check the tool's error text in the span output.
The same TOOL span many times with near-equal inputsThe agent is stuck. Add a step limit and look at the prompt before the loop.
A long gap with no child spanWork happens outside any traced code. Add a span around it.
A TOOL span with correct output, but a wrong final answerThe model misused the result. Look at the next LLM span's input.
The wrong tool chosen at the first LLM spanThe tool descriptions are unclear. Read the tool list the model saw.

Two habits help. First, read the failing run and one passing run of the same input side by side. The first different span is usually the cause. Second, when you fix a bug, keep the failing trace and turn it into a test case. MLflow's evaluation tools can score traces later, which closes the loop between debugging and regression tests.

What are the common mistakes when tracing agents?​

These five mistakes are common.

  • Enabling autolog after the client is created. Call autolog first, then build the agent or client.
  • Splitting one run across experiments. Distributed traces need the same tracking server and experiment. Otherwise they do not merge.
  • Tracing only the LLM. An LLM-only trace shows the model's tool request but not the tool's result. The tool span is where most failures show up.
  • Logging secrets. Tool inputs and outputs are recorded. If a tool handles passwords, tokens, or personal data, mask them before they reach the trace. MLflow has a masking guide.
  • Assuming a framework traces MCP. Some integrations do, as PydanticAI does. Others do not. Run one request and look for the tool span.

What about other tracing tools?​

You can do this with other tools. Langfuse, Arize Phoenix, Opik, OpenLIT, and OpenLLMetry all trace LLM applications, and they differ in license, hosting, and OpenTelemetry support. We compare them on one page: Best Open-Source LLM Tracing Tools in 2026. The idea in the six steps above (one trace per run, one span per step, one session ID per conversation) is not specific to MLflow, but the setup and API names differ by tool, so check each tool's docs.

For tracing multi-agent systems in particular, see our post on observability for multi-agent systems.

Can an agent read its own traces through MCP?​

Yes, in a different direction. MLflow also ships an MCP server that exposes trace operations to AI assistants. It can search and retrieve traces, analyze trace performance, log feedback, and manage tags. It is marked experimental and needs MLflow 3.5.1 or newer, installed with pip install 'mlflow[mcp]>=3.5.1'. See the MLflow MCP server docs.

This is separate from tracing your own MCP calls. It lets a coding assistant such as Claude Code look at the traces you already collected.

FAQ​

How do I trace AI agent activity end to end?​

Start an mlflow server, call your framework's autolog function (for example mlflow.openai.autolog() or mlflow.langchain.autolog()) before you create the client, and run one request. If the tree has a long gap with no child span, some work runs outside traced code. Wrap that code in @mlflow.trace and run the request again.

How do I trace MCP tool calls?​

First, run one request and look for a TOOL span under the LLM span that asked for the tool. If there is none, your framework does not trace MCP. Then put the client's call_tool call inside mlflow.start_span(name=..., span_type=SpanType.TOOL), so the span records the tool name, its arguments, and its result.

Can I trace an MCP server that runs in another process?​

For an HTTP server, yes. The client sends headers from get_tracing_context_headers_for_http_request, and the server reads them with set_tracing_context_from_http_request_headers. Both sides must log to the same tracking server and experiment, or the spans do not merge. A stdio server has no HTTP headers, so trace inside the server and use the same session ID to find its traces.

Do I need OpenTelemetry to trace an agent?​

No. You only need it when spans come from services that do not use the MLflow SDK. Those services set OTEL_EXPORTER_OTLP_TRACES_ENDPOINT to <tracking-uri>/v1/traces and pass x-mlflow-experiment-id=<experiment-id> in OTEL_EXPORTER_OTLP_TRACES_HEADERS.

How do I make agent behavior observable in production?​

Attach a user and a session to each trace (the session_id and user parameters, or the mlflow.trace.session and mlflow.trace.user metadata keys before MLflow 3.12.0), so you can filter by customer and conversation. Mask passwords, tokens, and personal data before they reach a span. When you fix a bug, keep the failing trace as a test case, so the same failure is caught next time.

Conclusion​

Before tracing, an agent bug is a guess. After tracing, it is a span you can open. Start small: run mlflow server, add the autolog call for your framework, and wrap each MCP call in a TOOL span. Run one request and read the tree.

Next step: copy the example from Step 3 into a file, run it against a local MLflow server, and open the trace in the UI. The MLflow Tracing overview shows what to add next.

Updated 2026-11-11.