Automatic Issue Detection
MLflow Issue Detection analyzes traces in an experiment to find recurring quality and operational problems. It inspects inputs, outputs, spans, tool calls, timing, errors, assessments, and metadata, then groups related findings into issues you can investigate and fix. For example:
- A customer-support agent invents a refund policy when answering a return request.
- A research agent repeats the same web search, slowing the response and increasing costs.
- A travel agent omits a required argument, causing a booking tool to fail.
- An extraction agent returns prose instead of the required JSON schema.
The following demo shows how to run detection, review evidence, open a source trace, resolve an issue, and schedule future runs. The analysis wait is shortened in the video.
How it works
Issue detection uses a multi-stage AI analysis pipeline:
Identify potential problems
Analyze sampled traces with your chosen model, focus areas, and detection priorities. Use conversation context when traces have session IDs.
Combine findings and context
Combine the model's findings with human feedback and recorded execution paths, grouping traces by session when available.
Group findings into issues
Group related failures, merge duplicates, and identify each issue's description, root cause, severity, and categories.
Match existing issues
Compare findings with existing Open and Not an issue entries. Reuse matching issues while preserving their status.
Attach evidence for review
Save new issues or refresh matching ones, and link affected traces with explanations and, when available, walkthroughs of relevant spans. Review the evidence in the Issues tab.
Requirements
- An MLflow experiment with recorded traces.
- An MLflow tracking server on Linux or macOS with a SQL database backend, such as SQLite or PostgreSQL, and background job execution enabled. See Self-hosting MLflow.
- Access to a supported model through a provider API key or an MLflow AI Gateway endpoint.
Start issue detection
- Open an MLflow experiment and select the Issues tab.
- Click Run issue detection. If the experiment already has issues, click Detect Issues instead.
- Choose a provider and model, such as OpenAI, Anthropic, or Google Gemini, or select an AI Gateway endpoint. Supply an API key when prompted, or use a saved credential.
- Review the selected traces and estimated cost, then click Run Analysis.
You can also start detection from the Traces tab. Filter or select traces there to analyze a specific subset. For multi-turn applications, record session IDs to preserve conversation context.
Analysis runs in the background. You can leave the page and return later.
Review issues
After the analysis finishes, the Issues tab lists detected problems and shows supporting evidence for the selected issue. Each piece of evidence links to an affected source trace. The issue details also show its description, severity, categories, and affected trace count.
Click View issues in the completion notification to return to the Issues tab.
Expand an entry in Evidence to read why that trace supports the finding. Evidence can include a walkthrough of the relevant spans, with highlighted inputs, tool results, and responses. Click Open trace to inspect the source trace in the drawer, or View span to jump to a particular span.

Triage issues
If a finding is not applicable, update the status to Not an issue using the dropdown in the upper-right corner. MLflow uses this feedback to avoid surfacing the same issue again in later detection runs.
Use the status dropdown in the upper-right corner of the issue details to change its status:
| Status | When to use it |
|---|---|
| Open | A newly detected issue that needs investigation. |
| Resolved | You have fixed the issue and verified the change. |
| Not an issue | The finding is a false positive or describes intentional behavior. |
Fix issues with coding agents
Each issue includes a fix prompt with the context and representative trace IDs a coding agent needs to investigate the problem.
- Install the MLflow skills in your coding agent.
- Configure the agent's environment to connect to the same MLflow tracking server, including any required authentication and workspace selection.
- In Fix this issue, click Show more to review the full prompt, then Copy.
- Paste the prompt into a coding agent running in your application repository. The agent can inspect the traces and code, propose a focused change, and validate the fix.
Review the proposed change before deployment. Evaluate the updated agent against representative cases to confirm the fix and check for regressions, then mark the issue as Resolved.
Schedule detection
Schedule issue detection to analyze new traces regularly:
- Click Schedule detection on the Issues tab.
- Choose Weekly or Daily. For a weekly schedule, select the day of the week.
- Set the time and timezone, then choose the model to use.
- Optionally, click Add filter to limit analysis by a metadata field or tag value, such as
environment = production. - Click Save. If prompted, save the provider credential so future runs can use it.

Scheduled runs require the server's periodic tasks and background job execution to be enabled. If Schedule detection is unavailable, check the server configuration. Each scheduled run incurs model usage costs.
Customize detection criteria
Click the Issue detection settings gear icon on the Issues tab to configure detection for the experiment:
- Detection priorities: Describe what matters to your application, such as "Treat incorrect refund promises as high severity" or "Ignore cosmetic formatting issues."
- Focus areas: Turn built-in categories on or off.
- Custom category: Add an application-specific category, such as "User frustration."

The built-in focus areas are:
| Focus area | Examples of findings |
|---|---|
| Correctness | Factual errors, hallucinations, or responses that contradict provided data. |
| Latency | Unusually slow responses or spans. |
| Execution | Failed tool calls, errors, or incomplete actions. |
| Adherence | Violations of instructions, constraints, or output formats. |
| Relevance | Responses that do not address the user's request. |
| Safety | Harmful, biased, or policy-violating output. |
| Cost | Unusual token or tool-call spending compared with the agent's baseline. |
Click Save to apply the settings to future detection runs. If no focus areas are selected, detection considers all of them.
View run history
Issue detection run history lists manual (Ad hoc) and Scheduled runs, including runs that find no issues or fail. You can open it in any of these ways:
- After starting detection: Click View progress in the notification.
- If detection fails: Click View details in the failure notification.
- If the Issues tab has no issues after a run: Click View run history beneath the Run issue detection or Start new detection button.
- If the Issues tab lists issues: Click the Issue detection run history icon beside the settings icon.
The table shows each run's status, analyzed trace count, new issue count, model, cost, and duration when available. Hover over a failed run's status to see its error details.
Window covered shows the trace time range for a scheduled run. It is empty for an ad hoc run, which analyzes a selected set of traces. When a schedule is active, the page also shows its cadence and next run time.
Access issues programmatically
Use mlflow.genai.create_issue(), mlflow.genai.search_issues(), and mlflow.genai.update_issue() to integrate issue triage with your own workflows.
Create an issue
Create an issue for a problem identified through manual review or another workflow:
import mlflow
from mlflow.entities import IssueCategory, IssueSeverity
mlflow.set_tracking_uri("http://localhost:5000")
experiment_id = "<experiment-id>"
created_issue = mlflow.genai.create_issue(
experiment_id=experiment_id,
name="Incorrect refund policy",
description="The agent promises refunds outside the documented 30-day return window.",
severity=IssueSeverity.HIGH,
categories=[IssueCategory.CORRECTNESS],
)
New issues default to Open (pending). The returned issue contains an issue_id you can use to retrieve or update it.
Search issues
The following example searches for open, high-severity issues in the same experiment:
issues = mlflow.genai.search_issues(
experiment_id=experiment_id,
filter_string="status = 'pending'",
severities=[IssueSeverity.HIGH],
max_results=100,
view="full",
)
for issue in issues:
print(issue.issue_id, issue.name, issue.description)
Search also supports name and category filters, sorting, and pagination. Pass issues.token as page_token to retrieve the next page when the token is present.
Update an issue
After verifying a fix, update the issue by ID:
from mlflow.entities import IssueStatus
mlflow.genai.update_issue(
issue_id=created_issue.issue_id,
experiment_id=experiment_id,
status=IssueStatus.RESOLVED,
)
The API status values pending, resolved, and rejected correspond to Open, Resolved, and Not an issue in the UI.
Cost and usage
Detection makes calls to your selected model. Cost depends on the number and size of traces, the model, and provider pricing. Review the estimate before starting and the recorded cost in run history afterward.
Start with a representative set of traces, then use trace filters and a suitable schedule to control recurring usage. If you use an MLflow AI Gateway endpoint, you can configure budget alerts and limits.