SIMBA Alignment Optimizer

MLflow provides a prompt-optimization alignment optimizer built on DSPy's implementation of SIMBA (Simplified Multi-Bootstrap Aggregation). Pass SIMBAAlignmentOptimizer to align() to use it instead of the default MemAlign optimizer.

Requirements

For alignment to work:

• Traces must contain human assessments (labels) with the same name as the judge
• Natural language feedback (rationale) is highly recommended for better alignment
• Minimum of 10 traces with human assessments required
• A mix of positive and negative labels is recommended

Basic Usage

See make_judge documentation for details on creating judges.

python

from mlflow.genai.judges import make_judge
from mlflow.genai.judges.optimizers import SIMBAAlignmentOptimizer
import mlflow

# Create a judge
judge = make_judge(
    name="politeness",
    instructions=(
        "Given a user question, evaluate if the chatbot's response is polite and respectful. "
        "Consider the tone, language, and context of the response.\n\n"
        "Question: {{ inputs }}\n"
        "Response: {{ outputs }}"
    ),
    feedback_value_type=bool,
    model="openai:/gpt-5-mini",
)

# Retrieve traces with human feedback
traces_with_feedback = mlflow.search_traces(return_type="list")

# Opt into SIMBA explicitly (the default optimizer is MemAlign)
optimizer = SIMBAAlignmentOptimizer(
    model="openai:/gpt-5-mini"  # Model used for optimization
)
aligned_judge = judge.align(traces_with_feedback, optimizer)

Debugging

To debug the optimization process, enable DEBUG logging:

python

import logging

logging.getLogger("mlflow.genai.judges.optimizers.simba").setLevel(logging.DEBUG)
aligned_judge = initial_judge.align(traces_with_feedback, optimizer)