# Tracing with OpenTelemetry

[OpenTelemetry](https://opentelemetry.io/) is a CNCF-backed project that provides vendor-neutral observability APIs and SDKs to collect telemetry data from your applications. MLflow Tracing is fully compatible with OpenTelemetry, making it free from vendor lock-in.

## Using the MLflow Tracing SDK[​](#using-the-mlflow-tracing-sdk "Direct link to Using the MLflow Tracing SDK")

The MLflow Tracing SDK is built on top of the OpenTelemetry SDK. If you want to instrument your AI applications with minimal effort, use the [MLflow Tracing SDK](/docs/latest/genai/tracing/quickstart.md).

python

```
import mlflow

from openai import OpenAI



mlflow.openai.autolog()



client = OpenAI()

response = client.responses.create(model="gpt-4o-mini", input="Hello, world!")
```

## Using Other OpenTelemetry Libraries[​](#using-other-opentelemetry-libraries "Direct link to Using Other OpenTelemetry Libraries")

You may want to trace LLMs or frameworks that are not [supported](/docs/latest/genai/tracing/integrations.md) by the MLflow Tracing SDK, or instrument applications written in languages other than Python and TypeScript/JavaScript.

MLflow Server exposes an OTLP endpoint at `/v1/traces` ([OTLP](https://opentelemetry.io/docs/specs/otlp/)) that accepts traces from any OpenTelemetry instrumentation, allowing you to trace applications written in other languages such as Java, Go, Rust, etc. To export traces to MLflow, set `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` to the MLflow server endpoint and set the `x-mlflow-experiment-id` header to the MLflow experiment ID.

bash

```
export OTEL_EXPORTER_OTLP_TRACES_ENDPOINT=http://localhost:5000/v1/traces

export OTEL_EXPORTER_OTLP_TRACES_HEADERS=x-mlflow-experiment-id=123
```

When [workspaces](/docs/latest/self-hosting/workspaces/getting-started.md) are enabled on your MLflow server, OTLP clients can also send `X-MLFLOW-WORKSPACE` to override the default workspace for a request.

For more details about the MLflow OpenTelemetry integration, see [Collect OpenTelemetry Traces into MLflow](/docs/latest/genai/tracing/opentelemetry/ingest.md).

## Combining the OpenTelemetry SDK and the MLflow Tracing SDK[​](#combining-the-opentelemetry-sdk-and-the-mlflow-tracing-sdk "Direct link to Combining the OpenTelemetry SDK and the MLflow Tracing SDK")

Since the MLflow Tracing SDK is built on top of the OpenTelemetry SDK, you can combine them to get the best of both worlds. To use both SDKs in a single application, set the `MLFLOW_USE_DEFAULT_TRACER_PROVIDER` environment variable to `false`. Then, you should invoke [`mlflow.tracing.set_destination()`](/docs/latest/api_reference/python_api/mlflow.tracing.html#mlflow.tracing.set_destination) to add MLflow's span processors to the existing trace provider, ensuring spans from both OpenTelemetry and MLflow to be combined into a single trace.

The following example shows how to combine MLflow's OpenAI auto-tracing with OpenTelemetry's native FastAPI instrumentation.

python

```
import os

import mlflow

from mlflow.entities.trace_location import MlflowExperimentLocation

from contextlib import asynccontextmanager

from fastapi import FastAPI, Request, Response

from openai import OpenAI

from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor



# Use the OpenTelemetry tracer provider instead of MLflow's default tracer provider.

os.environ["MLFLOW_USE_DEFAULT_TRACER_PROVIDER"] = "false"





# Enable MLflow OpenAI auto-tracing at application startup.

@asynccontextmanager

async def lifespan(app: FastAPI):

    mlflow.set_tracking_uri("http://localhost:5000")

    exp_id = mlflow.set_experiment("FastAPI").experiment_id

    # this ensures mlflow spans and FastAPI instrumentation spans are combined

    mlflow.tracing.set_destination(MlflowExperimentLocation(exp_id))

    mlflow.openai.autolog()

    yield





app = FastAPI(lifespan=lifespan)

# Enable FastAPI auto-instrumentation, which creates an OpenTelemetry span for each endpoint call.

FastAPIInstrumentor.instrument_app(app)





@app.post("/rag/v1/answer")

@mlflow.trace

async def answer_question(query: Request) -> Response:

    return ...
```

Spans generated from both SDKs will be merged into a single trace.

![The MLflow UI showing the MLflow and OpenTelemetry combined spans](/docs/latest/images/llms/tracing/opentelemetry/mlflow-otel-combined.png)

### Setting Up TraceProvider By Yourself[​](#setting-up-traceprovider-by-yourself "Direct link to Setting Up TraceProvider By Yourself")

You can explicitly configure your own `TracerProvider` (for example, to add custom processors or exporters). Setting `MLFLOW_USE_DEFAULT_TRACER_PROVIDER` to `false` and calling [`mlflow.tracing.set_destination()`](/docs/latest/api_reference/python_api/mlflow.tracing.html#mlflow.tracing.set_destination) ensures MLflow and OpenTelemetry spans are combined into a single trace.

important

[`mlflow.tracing.set_destination()`](/docs/latest/api_reference/python_api/mlflow.tracing.html#mlflow.tracing.set_destination) must be invoked after you explicitly configured the trace provider.

python

```
import os

import mlflow

from mlflow.entities.trace_location import MlflowExperimentLocation

from opentelemetry import trace as otel_trace

from opentelemetry.sdk.trace import TracerProvider



# Use the OpenTelemetry tracer provider instead of MLflow's default tracer provider.

os.environ["MLFLOW_USE_DEFAULT_TRACER_PROVIDER"] = "false"



# Set up the MLflow experiment

mlflow.set_tracking_uri("http://localhost:5000")

experiment_id = mlflow.set_experiment("my_experiment").experiment_id



# Create and set your own TracerProvider

external_provider = TracerProvider()

otel_trace.set_tracer_provider(external_provider)



# Set MLflow tracing destination to the experiment (or UC schema), this ensures mlflow

# span processors are added to the tracer provider

mlflow.tracing.set_destination(MlflowExperimentLocation(experiment_id))



# Now you can use both OpenTelemetry and MLflow tracing together

otel_tracer = otel_trace.get_tracer("my_app")



with otel_tracer.start_as_current_span("http_request") as external_span:

    external_span.set_attribute("http.method", "GET")



    # MLflow spans will be nested under the OpenTelemetry span

    with mlflow.start_span("model_prediction") as mlflow_span:

        mlflow_span.set_inputs({"query": "test"})

        mlflow_span.set_outputs({"result": "success"})
```

Both the OpenTelemetry span (`http_request`) and the MLflow span (`model_prediction`) will appear in the same trace in the MLflow UI.

![The MLflow UI showing the MLflow and OpenTelemetry combined spans when trace provider is set](/docs/latest/images/llms/tracing/opentelemetry/mlflow-otel-trace-provider.png)
