Skip to main content

One post tagged with "feature toggle LLM"

View All Tags

Engineers: Start LLM Feature Flags at 1% with Configs and MLflow

· 11 min read

Engineers reviewing an LLM rollout decision

Feature flags should be your LLM runtime control plane: use them to select models, cap tokens, route prompts, and auto-revert bad behavior without redeploying. Treat each flag as a structured decision point rather than a simple on/off switch, guided by NIST risk principles and tooling for tracing what happens next.

TL;DR:

  • Using feature flags for LLMs allows real-time control over model selection, cost caps, and prompt routing, reducing risks from model drift and unpredictability.
  • Starting with small traffic percentages and evolving from boolean to configuration-based flags enables safer rollout and better risk management across safety, cost, and performance metrics.
  • Robust reliability plumbing, including timeouts, retries, fallbacks, and detailed trace attribution, is essential to prevent outages and track regressions caused by model or prompt changes.
  • Continuous telemetry on latency, error rates, token consumption, and eval scores supports automated monitoring, canary testing, and instant rollback decisions for safer deployments.
  • Local flag evaluation with caching, standard conventions, and structured event logging provides lower latency, consistent deployment, and integrated observability within the control plane.

Table of Contents​

Why feature flags matter more for LLM-driven features​

A standard feature flag hides a button or a page. An LLM flag governs something far less predictable: a model that can drift in behavior between calls, cost wildly different amounts depending on prompt length, and fail in ways staging environments rarely surface. Testing a prompt change against a fixed set of examples tells you little about how it behaves against the long tail of real user input, so you need runtime levers that work after launch, not just before it.

That's what makes flags essential infrastructure here, not a nice-to-have:

  • Nondeterminism: the same prompt can produce different outputs across calls, so a passing test suite doesn't guarantee safe production behavior.
  • Cost exposure: a prompt tweak or model swap can quietly double your token spend per session.
  • Latency risk: routing to a slower or overloaded model degrades user experience without any code deploy.
  • Audit needs: regulators and internal reviewers increasingly expect a record of what model served what response, and why.

Flags, done right, carry that context automatically through structured events tied to every evaluation.

Feature-flag patterns every LLM team should implement​

Most teams start with booleans and quickly discover they need more. An LLM flag often needs to return a whole configuration, not just true or false. Here's a practical progression:

  1. Guardrail flags: return a config object specifying token cap, per-user rate limit, and allowed model, so one evaluation controls three risk dimensions at once.
  2. Prompt and policy flags: swap prompt templates or safety filters by audience segment or traffic percentage, letting you test a rewritten system prompt against 5% of sessions.
  3. Kill switches: a single flag that instantly reverts an entire feature to a known-safe model and prompt combination when something breaks.

Booleans still work for simple feature gates (show or hide a chat widget), but anything touching cost, safety, or model choice deserves a config object. A config object lets you change five parameters in one evaluation instead of coordinating five separate flags that can drift out of sync.

Rollout granularity matters as much as the flag type. Segmenting by user cohort, region, or account tier gives you a controlled blast radius if a new model underperforms.

Pro Tip: Start new LLM flags at 1% traffic with a config object, not a boolean. It costs nothing extra and saves a rewrite later.

Runtime reliability plumbing for every flagged LLM call​

A flag that routes traffic to a new model is only as safe as the reliability code behind it. Without timeouts and fallbacks, a single slow provider can cascade into an outage, and a production checklist for deploying LLMs lists this plumbing as a baseline requirement, not an advanced feature.

Build these in before you flip any model-routing flag live:

  • Timeouts and retries: apply exponential backoff with jitter so retries don't stampede an already struggling provider.
  • Circuit breakers: stop sending traffic to a model that's failing repeatedly, and fail fast instead of queueing requests.
  • Multi-provider fallback: let a flag define a primary and secondary model, so a provider outage triggers an automatic switch instead of a manual scramble.
  • Full attribution in traces: every call should log model version, prompt version, token counts, and cost, so a regression traces back to the exact flag state that caused it.

Pinning a model and prompt version behind a flag, with one-click rollback to the last known-good combination, is one of the highest-leverage defenses against a silent regression slipping through, as that same deployment checklist notes.

Observability, eval gates, and automated canary checks​

None of this works without telemetry wired directly into the flag evaluation itself. You need to know, per flag variant, what happened to latency, cost, and quality, not just whether the code ran.

Track at minimum:

  • Latency per model and prompt version.
  • Error rate, including timeouts and provider-side failures.
  • Token consumption and cost per session, since these can spike independently of error rate.
  • Eval score from an automated judge or rubric, tracked per flag variant.

Feature flags have evolved into an AI-native control plane: Datadog describes how a single flag can gate a model, throttle a token budget, and emit structured events for downstream analysis, all at once.

A canary contract turns those metrics into an automatic decision. Define a baseline from the current production variant, set thresholds for each metric (say, error rate under 2% and eval score within 5 points of baseline), pick a sample size large enough to avoid noise, and set an observation window before ramping further. Datadog recommends automating this ramp-or-revert logic directly off flag-evaluation telemetry rather than waiting for a human to notice a dashboard trend. Emitting a structured event on every flag evaluation, tagged with the flag key and variant, lets your analytics layer join outcomes back to the exact decision that produced them.

Choosing SDKs, local evaluation, and standards for flag delivery​

Where you evaluate a flag matters almost as much as what it decides. Remote evaluation, calling out to a flag service on every request, adds a network round trip to every LLM call, which is the last place you want extra latency. Local evaluation, where rules are downloaded and cached on the client or edge, removes that round trip entirely.

Local and remote flag evaluation paths

Cloudflare's Flagship is a clear example: binding flag evaluation into edge workers avoids a remote call and keeps rollout bucketing consistent without per-request overhead.

A few practical considerations when you set this up:

  • Cache local rule sets with a short TTL so updates propagate within minutes, not hours.
  • Use consistent hashing on a stable user ID so the same user always lands in the same variant.
  • Adopt OpenTelemetry's feature flag conventions for your structured events, which standardizes keys like flag key and evaluation reason across tools.
  • Enforce lifecycle hygiene: every flag gets an expiry date, an owner, and a naming convention, or it becomes permanent technical debt.

OpenFeature-compatible SDKs are worth prioritizing here, since they let you swap providers later without rewriting every call site.

How MLflow supports LLM feature-flag workflows​

Tracing tools map naturally onto this whole pattern. Each LLM call can carry model version, prompt, token counts, and cost, so attaching a flag key and variant to that trace can turn your flag evaluations into first-class, queryable data rather than a separate log you have to stitch together later.

In practice, a team might combine MLflow with their flag provider like this:

  • Log the active flag variant as a trace attribute on every LLM call, alongside model and prompt version.
  • Use an AI Gateway to centralize prompt versions and routing rules, so a flag change at the gateway level can propagate consistently across services.
  • Run evaluation tooling, including LLM-as-a-judge scoring, against canary traffic to generate the eval score that feeds your automated ramp or revert decision.
  • Keep an audit trail of which flag state produced which trace, satisfying governance reviews without manual reconstruction.

Thinking of flags as the delivery layer for LLMs​

The teams that get this right treat observability and flag lifecycle policy as the foundation, then layer automation on top, never the reverse. Automating a canary revert before you have reliable eval scores just automates the wrong decision faster.

The most common mistake I see is a flag hardcoded into a conditional with no expiry and no owner. It works fine until the model behind it changes behavior six months later, and nobody remembers the flag exists.

— Kevin

MLflow as a foundation for your LLM control plane​

MLflow brings the pieces this article recommends into one open-source platform: tracing that captures flag context alongside model and cost data, an AI Gateway for centralized prompt and routing control, and evaluation tooling built for automated canary gating.

Mlflow

Explore the MLflow platform to see how these pieces fit together for your own deployment pipeline.

FAQ​

What are examples of feature flags for LLM features?​

Common examples include a flag that caps tokens per session, one that routes traffic between two model providers, and one that swaps a prompt template for a test cohort. Config-object flags that bundle model choice, rate limit, and safety filter into a single evaluation are increasingly common for LLM features.

Should I turn on feature flags for every LLM feature?​

Any LLM feature touching cost, safety, or model choice benefits from a flag, since it gives you a way to revert without a deploy. Simple, low-risk UI elements can often ship without one, but anything calling a model in production should have a kill switch available.

Should you remove feature flags once a rollout is complete?​

Yes, a flag with no expiry date becomes technical debt that complicates future debugging and audits. Setting a TTL and an owner at creation, as recommended in flag lifecycle best practices, makes cleanup routine rather than optional.

What happens if I turn off all feature flags at once?​

Every flagged feature reverts to its default configuration, which should be the last known-safe model, prompt, and rate limit combination. This is exactly the behavior a well-designed kill switch relies on, so defaults need to be tested and kept current, not just assumed safe.

Sources​