Skip to main content

Token Budgeting Strategies for LLM Apps and Agents

· 15 min read

Engineer configuring an application token budget

A token budget is an application-level limit on the input, output and multi-call tokens a workflow may spend. The strategy that holds up in production has four parts. Split the budget by prompt component. Cap every agent loop. Measure cost per completed task, not per request. Gate each saving with an evaluation, so cheaper never quietly means worse. Tracing and evaluation in one place make each of those steps measurable. That is the case for running them on MLflow, where the lifecycle of GenAI and agent applications lives in one open-source platform.

TL;DR​

  • Budget by component (system prompt, tools, retrieval, history, current turn, output) so the parts sum to an input target derived from your latency and cost goals.
  • Budget the task, not the request. Retries, agent iterations, tool follow-ups and summarization all belong in the task's total.
  • Typical savings from comprehensive optimization are 20 to 40%, but compression can cost quality, so every cut needs an evaluation gate.
  • Use static caps by default. Move to adaptive budgets only when task difficulty varies and you have a reliable difficulty signal.
  • Track tokens per component, per agent and per task type from day one, and set spend limits before an agent loop finds its own.

Table of Contents​

What a token budget is, and why the context window is not one​

The context window is what the model can hold. A token budget is what your application is willing to spend. Those are different numbers, and mixing them up is the most common budgeting mistake.

A request can fit comfortably in a provider's window and still be too expensive, too slow or too noisy. More input generally raises billed cost and processing latency. Excessive context can also dilute the model's attention and leave too little room for the answer. Budgeting therefore sits on four axes at once: cost, latency, context quality and output completeness.

Four axes of token budgeting

For a single chat completion, the budget is simple arithmetic. For an agent, it is not. An agent resends its system prompt, tool definitions and prior turns on every loop iteration, so a run of twelve steps does not cost twelve times a single prompt. It costs more, because the history grows as it goes. This is why the strategies below are organized around the workflow, not the call.

Strategy 1: Decompose the prompt and budget each component​

Start by breaking every prompt into the parts that compete for space. AWS's agentic AI guidance describes this decomposition as system instructions, tools, retrieved content, conversation history and the current turn. It then allocates a budget to each component so that the budgets sum to an input target derived from your latency and cost objectives, and measures actual usage per component.

A worked example makes this concrete. Say your support agent has an input target of 8,000 tokens per model call, derived from a latency goal. One plausible split:

ComponentBudget (tokens)Policy when exceeded
System instructions800Fixed. Fail the deploy review if it grows.
Tool definitions1,200Load only task-relevant tools
Retrieved context3,000Re-rank, keep top passages
Conversation history2,000Summarize older turns
Current user turn1,000Truncate with a notice
Total input8,000

The numbers are illustrative, yours will differ. The point is that each row has an owner and a policy. When a regression pushes the average call to 11,000 tokens, you can see which row grew instead of arguing about the whole prompt.

Once each component has a budget, the capacity you buy to serve it matters too. Teams with predictable volume sometimes look at reserving AI inference capacity so that the latency target behind the input budget is backed by a performance requirement they have chosen.

Strategy 1: Decompose the prompt and budget each component

Strategy 2: Trim the biggest components first​

The FinOps Foundation's guidance on GenAI usage lists the main techniques: prompt design, prompt compression, context management, output control, compact formats, strategic truncation and caching. It states that comprehensive token optimization typically reduces consumption by 20 to 40% with minimal impact on response quality. Treat that as a planning range for what is achievable across a stack, not a promise for any one application.

For agents, four levers do most of the work.

Tools: load only what the task needs​

Injecting a full tool catalogue into every call is expensive and distracting. AWS recommends dynamically selecting only task-relevant tools, keeping tool schemas compact, and using semantic tool retrieval instead of the full catalogue. If you connect many Model Context Protocol servers, tool definitions can become the largest fixed cost in the prompt, so audit them first.

History: bound it​

A raw transcript kept indefinitely grows without limit. AWS recommends bounding history with a summary of older turns plus a sliding window of recent ones. Budget the summary itself, and count the tokens the summarization call consumes against the task.

Caching: make repeated prefixes cheaper​

Anthropic's cost optimization guidance recommends prompt caching for growing agent conversations, because every turn resends system prompts, tool definitions and prior turns. Cached prefixes get a lower cache-read price. They still occupy context capacity, so caching reduces your bill but does not relieve your context budget. Budget those two things separately.

Output: control length deliberately​

Output tokens are a budget line, not an afterthought. Set explicit length expectations in the prompt, use structured or compact formats where downstream code consumes the result, and set a sensible max_tokens. When runs end with stop_reason: max_tokens, Anthropic's guidance is to raise the limit to 64,000 for those attempts, so a cap that is too tight is itself a cost: a truncated answer gets retried.

Strategy 3: Gate every saving with an evaluation​

Compression is where budgeting turns dangerous. A study of minification for state-in-context software agents reported 42% lower average input-token use, but also a 12-percentage-point drop in resolution rate. A cheaper agent that solves fewer tasks is not cheaper per solved task.

The fix is procedural. Never ship a budget change without running the same evaluation set before and after, and track a quality metric next to the token metric. In practice:

  1. Build a representative dataset of real tasks, including hard ones.
  2. Score outputs with an automated judge plus a few deterministic checks (schema validity, tool-call correctness).
  3. Compare the old and new budget on resolution quality, tokens per task and latency together.
  4. Ship only if quality holds within a threshold you set beforehand.

This is where LLM-as-a-Judge evaluation earns its place in a budgeting workflow. It turns "the summary seems fine" into a score you can compare across budget variants. Our tracing captures what each agent step actually sent and received, so when a score drops after a cut, you can open the traces and see which truncated context caused it.

Strategy 4: Budget the task, not the request​

A cheap request can hide an expensive workflow. A production measurement should count retries, agent iterations, tool follow-ups and summarization calls inside the task's budget. Otherwise the dashboard shows a healthy average cost per call while cost per resolved ticket climbs.

The unit that closes the loop is cost per completed task:

cost_per_task = sum(input, output, cache-read, cache-write tokens x price)
over every model call in the task, including retries

To get it, every model call in a run must carry the same task identifier, which is exactly what a trace gives you: a parent run with child spans for each model call and tool call. Group by task type, and you can finally compare a refund flow against a password-reset flow and put the budget where the cost actually is.

Set per-task ceilings​

For each task type, define a soft limit (log and alert) and a hard limit (stop or hand off). A minimal guard looks like this:

class TaskBudget:
def __init__(self, soft, hard):
self.soft, self.hard, self.used = soft, hard, 0

def charge(self, tokens):
self.used += tokens
if self.used >= self.hard:
raise RuntimeError("Token budget exhausted: hand off or summarize")
return self.used >= self.soft # True means: warn and degrade

Call charge after every model response with the usage the API returns. What happens at the hard limit is a product decision, so agree on it in advance: return a partial answer, escalate to a human, or switch to a cheaper model for the final step.

For costly agent loops, Anthropic also recommends setting a session budget and a workspace spend limit on the provider side. Treat those as a backstop beneath your own per-task limits, not a replacement.

Strategy 5: Static budgets by default, adaptive where difficulty varies​

Static caps are predictable, easy to alert on and easy to reason about. Start there. Adaptive budgets pay off when difficulty varies a lot between tasks or turns and you have a reliable signal for that difficulty.

The research is encouraging for those cases:

  • A learned three-mode reasoning policy cut mean response length from 4,796 to 2,811 tokens, a 41% reduction, while staying close to baseline accuracy on MATH500. Its default caps were 1,024 tokens for the no-think mode and 3,000 for the short mode.
  • The TAB multi-turn research reports up to 35% token savings and 30% lower latency at maintained accuracy by giving easier turns smaller budgets and saving capacity for harder later turns. A variant that sees the full planned sequence of subquestions reached up to 40% savings, which suggests that knowing the task's structure up front improves allocation.

These are research results on specific benchmarks. Your gain depends on your task mix, so the practical route is the same as before: run the adaptive policy against your evaluation set and compare tokens, latency and quality together.

A simple, production-friendly version of adaptivity does not need a learned policy. Route by task class: short factual lookups get a small output cap and a small model, multi-step investigations get a larger session budget. Then review the class boundaries monthly against your traces.

Strategy 6: Instrument everything with bounded dimensions​

You cannot budget what you do not measure. AWS recommends recording, on every request, token usage per component alongside input, output and cache-read and cache-write tokens, tagged with dimensions such as agent ID and task type.

Two practical rules keep this data useful:

  • Store raw counts, derive dollars later. Token counts do not change, prices do. Keeping raw input, output and cached counts and applying a price table at query time means a provider price change does not corrupt your history.
  • Keep dimensions bounded. Model, operation, feature and task type are good labels. User IDs, request IDs and raw prompt text blow up metric cardinality and should live in traces, not in metric labels.

Our own write-ups on token usage tracking and LLM cost tracking go deeper on the monitoring side, and the posts on latency budgets for LLMs cover the other half of the constraint, since a token budget is also a latency budget.

If several teams call several providers, a central AI Gateway is a natural place to apply limits and governance across providers, instead of re-implementing caps inside every service.

A 30-day rollout for a team​

You do not need to do everything at once. A sequence that works:

  1. Week 1: measure. Trace every model call with a task ID, an agent ID and a task type. Record per-component token counts where you can.
  2. Week 2: baseline. Compute cost per completed task and the p50 and p95 token counts per task type. Find the top two components by size.
  3. Week 3: cut and evaluate. Trim the largest component (usually tools or history), and run your evaluation set before and after. Keep only changes that hold quality.
  4. Week 4: guard. Add soft and hard per-task limits, agree on the degradation path, and alert on burn rate (tokens per hour per task type) rather than on a monthly balance.

Burn rate matters because a runaway loop can spend a month's allowance in an afternoon. A monthly total tells you about it after the fact. An hourly rate tells you while it is happening.

Perspective​

Most token budgeting advice starts from the invoice. We think it should start from the trace. An invoice tells you that you spent too much. A trace tells you which step of which agent run sent a 3,000-token tool catalogue to answer a one-line question.

That is why we treat tracing, evaluation and gateway governance as one lifecycle instead of three products. You find the expensive step in a trace, you test the cheaper version with an automated judge against real tasks, and you enforce the limit in the gateway so it applies to every provider and every team. Each stage hands evidence to the next.

MLflow is open source under Linux Foundation governance, and every feature is available without an enterprise paywall. Tracing builds on OpenTelemetry, so the data fits the tooling and frameworks you already use. Budgeting is a discipline that should not depend on which vendor's dashboard you happen to have.

Kevin

Put the budget on a platform you can inspect​

If you are building this loop now, we built MLflow for it: observability with deep tracing of agentic reasoning, automated LLM-based evaluation, prompt management and versioning, and a cross-provider AI Gateway, in one open-source platform that integrates with the major AI and ML frameworks.

Mlflow

Start with tracing on one agent, find its heaviest component, and let the data pick your first cut.

FAQ​

What is the first token budgeting step for an existing agent?​

Instrument it. Record input, output and cached tokens per model call, tagged with an agent ID and a task type, and group calls into tasks. Without cost per completed task you will optimize the wrong component.

How much can token optimization save?​

The FinOps Foundation states that comprehensive optimization typically reduces token consumption by 20 to 40% with minimal impact on response quality. Results vary by application, so measure savings and quality together on your own evaluation set before rolling out any change.

Should I use static or adaptive token budgets?​

Start static, because it is simple to monitor and alert on. Move to adaptive allocation when task or turn difficulty varies widely and you have a reliable difficulty signal. Research on multi-turn reasoning reports savings of up to 35% in that setting.

Does prompt caching remove the need for a context budget?​

No. Cached prefixes are billed at a lower cache-read price, but they still occupy context capacity. Budget cost and context space as two separate constraints.

Sources​