SIMBA Alignment Optimizer

MLflow provides the default alignment optimizer using DSPy's implementation of SIMBA (Simplified Multi-Bootstrap Aggregation). When you call align() without specifying an optimizer, the SIMBA optimizer is used automatically.

Requirements

For alignment to work:

• Traces must contain human assessments (labels) with the same name as the judge
• Natural language feedback (rationale) is highly recommended for better alignment
• Minimum of 10 traces with human assessments required
• A mix of positive and negative labels is recommended

Basic Usage

See make_judge documentation for details on creating judges.

python

from mlflow.genai.judges import make_judge
from mlflow.genai.judges.optimizers import SIMBAAlignmentOptimizer
import mlflow

# Create a judge
judge = make_judge(
    name="politeness",
    instructions=(
        "Given a user question, evaluate if the chatbot's response is polite and respectful. "
        "Consider the tone, language, and context of the response.\n\n"
        "Question: {{ inputs }}\n"
        "Response: {{ outputs }}"
    ),
    feedback_value_type=bool,
    model="openai:/gpt-5-mini",
)

# Retrieve traces with human feedback
traces_with_feedback = mlflow.search_traces(return_type="list")

# Default: Uses SIMBA optimizer automatically
aligned_judge = judge.align(traces_with_feedback)

# Explicit: Same as above but with custom model specification
optimizer = SIMBAAlignmentOptimizer(
    model="openai:/gpt-5-mini"  # Model used for optimization
)
aligned_judge = judge.align(traces_with_feedback, optimizer)

Default Optimizer Behavior

When using align() without an optimizer parameter, MLflow automatically uses the SIMBA optimizer. This simplifies the alignment process while still allowing customization when needed.

Debugging

To debug the optimization process, enable DEBUG logging:

python

import logging

logging.getLogger("mlflow.genai.judges.optimizers.simba").setLevel(logging.DEBUG)
aligned_judge = initial_judge.align(traces_with_feedback, optimizer)