LLM Cost Optimization: A Practical Guide for Agent Teams

LLM cost optimization is the practice of lowering what you pay per successful outcome, not per token. The levers that matter most are measurement, model routing, caching, context and output limits, and benchmarking each change against a fixed set of representative tasks. This guide orders those levers by the sequence in which they usually pay off, with particular attention to agent-powered products, where one user request can fan out into many model calls and tool calls.

TL;DR

Measure first. Attribute spend to a user, session, feature, model and provider, and track cost per accepted output rather than the token price alone.
Route by difficulty. Send simple steps to smaller models, escalate to stronger models only when a verifier or failure signal justifies the extra cost.
Cache and trim. Reuse stable prompt prefixes through provider prompt caching, compress context, and cap output length.
Benchmark every change on a fixed dataset that includes difficult and failure-prone requests, and keep a rollback path.
For agents, optimize the whole workflow (retries, tool calls, context growth), because per-token savings can be erased by extra steps.

Where LLM costs actually come from

The invoice from a model provider is only the visible part. A 2026 paper on inference unit economics lists the wider set: provider tokens, retrieval, orchestration, tool calls, retries, evaluation, human review, observability, networking, capacity and support. Each of these can be reduced, but only if it is first visible.

Token pricing itself is uneven. A FinOps Foundation whitepaper notes that, at the time of writing, cutting-edge models cost USD 10 to 20 per million generated tokens, that tokenization differs between models, and that input and generated tokens can be priced differently. Two consequences follow:

A price-per-million comparison between models is only approximate, because the same text produces a different token count under each tokenizer.
Output-heavy workloads (code generation, long reports) are exposed to the higher generated-token rate, while retrieval-heavy workloads are exposed to the input rate.

Why agents change the arithmetic

A single chat completion has one input and one output. An agent session loops: it reads context, calls a tool, appends the result, and calls the model again with a larger context. Spend therefore grows with the number of steps and with context length at each step. Retries after a schema failure or a tool error repeat the entire cost of a step. Our own write-up on why Claude Code gets expensive covers how this compounding appears in a coding agent and which changes actually reduce the bill.

Step 1: Measure before you optimize

Most cost surprises come from measurement gaps rather than from usage itself. Langfuse's guidance on LLM cost management calls for tracking spend at the token level, attributing it to specific users, sessions or features, and setting alerts. Attribution is what turns a monthly total into a decision, because it shows which feature or customer produced the spend.

A workable minimum for each request record:

Model and provider identifiers.
Input, cached-input and output token counts.
Product feature, workflow and customer or tenant identifiers.
Number of retries, tool calls and escalations.
Outcome: accepted, rejected, corrected manually, or failed.

The inference unit economics paper recommends measuring cost per accepted output, including the effects of retries and manual correction. This is the metric to keep in view throughout the rest of this guide. A model that costs half as much per token but needs two attempts and a human fix is not cheaper.

For self-hosted inference, infrastructure cost needs the same treatment. OpenCost's inference cost tracking documentation describes attributing infrastructure costs to models and namespaces, distributing shared-infrastructure costs across models, and calculating blended and differentiated input and output costs per million tokens, with KV-cache savings reported separately.

Step 2: Route each request to the right model

Sending every request to one frontier model is the most common source of avoidable spend. A survey of LLM routing published in the 2026 Findings of EMNLP explains that models differ substantially in price, latency and capability, and that routing turns model choice into a per-query decision that can move a deployment toward its achievable quality-cost frontier.

What the research shows

A University of British Columbia study of difficulty-aware hybrid routing found up to 40% fewer calls to the large model with no drop in response quality. One configuration routed 22% of queries to Llama 2 13B with less than a 1% BART-score decline.
The RouteLLM paper reports that routers trained on preference data reduced costs by more than 2x in some benchmark settings without substantially compromising response quality.

These are results on specific benchmarks and model pairs from 2024. Treat them as evidence that the approach works, not as a savings forecast for your workload.

Three routing patterns

Harvard's CS2680 course notes on cost-effective agents distinguish fixed cheap-versus-strong routing, cascades and budget-aware controllers. The differences matter in practice:

Pattern	How it decides	Best suited to	Main risk
Fixed routing	Static rule by task type (for example, classification goes to the small model)	Workflows with well-separated step types	Rules go stale as traffic changes
Cascade	Try the cheaper model first, escalate when a verifier or confidence signal says the stronger model is worth the cost	Tasks with a cheap automatic check	Verifier cost and added latency on escalated requests
Budget-aware controller	Allocates a spend budget across steps, retries and workers	Long-running agent sessions	More complex to build and tune

The same notes make a point that is easy to miss: the right target is the incremental benefit of a stronger model, an extra worker or a retry relative to its added cost, not merely whether a prompt looks difficult. Early weak actions in an agent can alter the later trajectory and retry behavior.

Practical escalation signals

According to a 2026 guide on cutting token costs, routing can escalate on schema failures, tool errors, policy flags or task-specific classifier results, and teams should track escalation rate, retries, latency and task success. Set thresholds against an explicit quality floor and risk level, not against a hunch.

If you want the mechanics of the router itself, our practical guide to LLM routers goes through how model routing is structured. Per-model rates for models that HarnessRouter supports are published on pages such as glm-5.3-flash pricing, llama-4-maverick pricing and llama-3.3-70b pricing, which makes it straightforward to compare candidates for a cheaper tier.

Step 3: Right-size models by step

Routing works best when it is designed per workflow step. OpenAI's agent optimization cookbook recommends this sequence:

Establish a baseline with a capable model on identical labeled tasks.
Replace models step by step: a smaller model for classification and extraction, a mid-tier model for routine workflows, and the strongest model for complex or high-risk cases.
Evaluate correctness, policy compliance, latency and cost per successful outcome at every substitution.

The baseline matters. Without a capable-model reference, you cannot tell whether a cheaper configuration lost quality or merely changed style.

Step 4: Cache, compress and cap

Once the right model handles each step, reduce the tokens each step consumes.

Prompt caching

Provider prompt caching bills repeated, stable prompt prefixes at a lower rate, and discounts and minimum prompt sizes vary by model. To benefit, structure prompts so that the static portion (system instructions, tool definitions, reference material) comes first and the variable portion (user input, latest tool result) comes last. A single reordering of a prompt can decide whether the cache is hit at all.

Compression and semantic caching

The April 2026 Local-Splitter measurement study of seven tactics describes prompt compression as shortening cloud input, semantic caching as serving similar requests without a cloud call, and vendor prompt caching as discounting repeated stable prefixes. In its tests, local routing plus prompt compression achieved 45 to 79% savings on edit-heavy and explanation-heavy workloads, and the full tactic set achieved 51% on RAG-heavy workloads. Its most actionable finding is that the best combination is workload-dependent.

Compression condenses context while a separate cache retrieves a reusable prior result.

Semantic caching deserves care. The guide above advises treating it as product behavior that needs freshness and privacy controls: a cached answer to a similar question can be wrong for a different user, or out of date.

Output limits and context hygiene

Set a maximum output length per step and request structured outputs where a downstream parser expects them, so that verbose prose is not generated and then discarded.
Trim tool results before they re-enter the context. A file listing or log excerpt rarely needs to be appended in full.
Summarize or drop stale turns in long sessions, since every later step pays for them again.

Infrastructure-level levers for self-hosted models

If you run models yourself, memory is often the constraint. NVIDIA's inference optimization overview notes that KV-cache memory grows linearly with batch size and sequence length, which limits throughput for long-context workloads. Batching and KV-cache management are therefore the main infrastructure levers, and they interact directly with the context-trimming above.

Step 5: Benchmark every change

Every lever above can weaken quality, resilience or margin. The inference unit economics paper frames cost as a holistic problem and recommends benchmarking representative production tasks for cost, latency, quality, failures and review, using documented rubrics, including difficult cohorts, and revalidating after version changes.

A benchmark that a team can trust has these properties:

Fixed dataset. Representative requests, plus expensive, difficult and failure-prone ones, so that a router cannot look good by handling only easy traffic.
Rubric before results. Define what counts as accepted before seeing outputs.
Several metrics at once. Cost per accepted output, median and tail latency, failure rate, fallback rate and human review rate.
Re-runs on version changes. A silent provider model update can shift both quality and token counts.

Building and maintaining this evaluation harness is real engineering work, and deciding whether to build it in-house is a total-cost question in its own right. Product leaders weighing that decision can read a breakdown of build versus buy software costs before committing engineers to it.

Rolling out a routing change safely

Operational guidance on canary testing candidate routes suggests comparing the candidate against a contemporaneous control, preserving assignment at user or conversation level so one conversation does not switch models midway, reporting median and tail latency, metered cost, invalid-run and fallback rates, and keeping an automatic rollback path. Start with a small share of traffic and widen only when cost per accepted output improves without the quality metrics moving.

Optimizing agent-powered products

For agents, the OpenAI cookbook's summary is to optimize the workflow rather than the single call: smaller models for simple steps, stronger models for high-risk cases, less unnecessary prompt, context, output and tool work, prompt caching, and evaluation of quality, latency, tool use and total cost on a representative set.

A retry loop adds work before the successful output reaches the acceptance gate.

In practice, the agent harness (the runtime that wraps the model with tools, sandboxing, session storage and context handling) is itself a cost variable. Two harnesses running the same underlying model can consume very different numbers of tokens on the same task because they manage context and tool calls differently. That makes harness choice a benchmarking question, not a default. Our agent harness benchmark compares Codex, Claude Code and Hermes, and integration pitfalls that inflate spend early are covered in what breaks on the first try.

Our perspective

Cost work fails most often when it is done one provider at a time. Each model vendor exposes its own session format, its own file handling and its own caching behavior, so every experiment with a cheaper model or a different harness becomes an integration project. Teams then stop experimenting, and the spend stays where it started.
We built HarnessRouter on the opposite premise: if switching a harness or a model is a configuration change behind one Agent API, then benchmarking becomes routine, and routine benchmarking is what finds the savings. This is why we publish the open-source Community Edition and the cloud platform from the same interface, and why our stated position is that model provider costs are passed through without markup, with the platform fee shown separately.
Our view is that the useful unit is cost per successful task, measured on your own traffic, across more than 30 supported harnesses. A single API for those harnesses lets a team run the same task set through several of them and read the results side by side, instead of arguing from vendor claims.

Running this on HarnessRouter

HarnessRouter is an open-source unified interface that lets product teams connect Codex, Claude Code, Hermes and other supported agent harnesses to their product through one Agent API. It includes benchmarking tools, session management, file handling, plugin support and environment provisioning, so the measurement and comparison steps above do not require separate backends for each harness.

The published pricing is structured so that you can start measuring before you commit:

Plan	Price
Free	$0 per month
Developer	$20 per month
Production	$200 per month
Scale	$1,000 per month
Enterprise	No published price

Additional fees listed on the pricing page include tracing at $50 per month, a 5.5% top-up credit processing fee. When you bring your own provider API keys, routing is fee-free up to $25,000 per month of model usage at listed prices, with a 5% routing fee above that volume. Agent memory (Beta) is listed at $0.0000840 per hour per GB.

If you prefer to run it yourself, the Community Edition is the open-source, self-hosted option. To see how the harnesses are exposed, start with the Codex harness or the Claude Code harness, or go to [harnessrouter.ai](/) for the platform overview.

A prioritized order of work

Because the best tactic combination depends on the workload, a fixed checklist overpromises. This order, however, tends to avoid wasted effort:

Instrument requests and compute cost per accepted output for each feature.
Fix structural waste: prompt ordering for caching, unbounded outputs, oversized tool results.
Introduce routing on the highest-spend, lowest-risk step, using a cascade with a cheap verifier.
Benchmark on a fixed dataset, canary the change, and keep a rollback path.
Repeat for the next step, and re-run benchmarks whenever a model or harness version changes.

FAQ

What are the main pillars of LLM cost optimization?

A 2026 guide on cutting token costs names four: efficient token usage through caching and compression, model selection and routing, infrastructure-level optimization such as batching and KV-cache management, and evaluation of cost per successful task. Measurement underlies all four, since none can be tuned without request-level data.

How much does one million tokens cost?

It depends on the model and on whether the tokens are input or output. The FinOps Foundation whitepaper reports USD 10 to 20 per million generated tokens for cutting-edge models at the time it was written, with input tokens priced differently. Check the current rate card of your provider, and note that tokenizers differ, so the same text yields different token counts.

Which LLM is the most cost-effective?

No single model wins across tasks. The cheapest model per token can be the most expensive per accepted output if it triggers retries or manual fixes. Establish a capable-model baseline, then test smaller models step by step and compare cost per successful outcome on your own labeled tasks.

Can routing hurt quality?

It can if thresholds are loose or the router is not tested on hard cases. Published results show routing with no drop in response quality under specific benchmark conditions. Set an explicit quality floor, include difficult requests in your test set, and roll out through a canary with automatic rollback.

Search the site

LLM Cost Optimization: A Practical Guide for Agent Teams

TL;DR