LLM inference cost is what you pay each time a trained model serves a request: token charges when you call a hosted API, GPU-hours when you run the model yourself, plus the storage, networking and platform overhead around both. It is a variable cost that grows with usage, so it has to be measured per request and per outcome, not per invoice. Teams that route traffic through one Agent API such as HarnessRouter can compare harnesses and models on the same task before committing to a price curve.
TL;DR
- Inference cost is an operating expense that scales with request volume, context length and output length, unlike training, which is an upfront cost for a particular training run rather than a charge for each inference request.
- For hosted APIs, cost per request is input tokens times the input rate plus output tokens times the output rate. Output tokens often cost more than input tokens; the ratio varies by model and provider.
- For self-hosted models, utilization decides the price per token: the same GPU bill is divided across however many tokens the hardware actually produces.
- The largest levers are model selection, prompt caching, batch processing, shorter outputs and measuring cost per successful outcome instead of cost per call.
- Benchmark candidate models and harnesses on your own tasks before you set a budget, and track cost per task continuously afterward.
What LLM and AI inference costs include
The terms "LLM inference costs" and "AI inference costs" describe the same thing at different scopes. AI inference cost is the expense of running a trained model to produce responses, predictions or classifications. For a managed model it appears as token or API charges. For a self-hosted model it appears as GPU compute, along with storage, networking and supporting cloud services.
A per-million-token price compresses a longer list. In practice the cost covers accelerator time spent generating each output, the memory that holds the model, the networking that moves requests, and the platform overhead that keeps endpoints available. When you read a headline price, remember that it is a summary of all of these.
Training differs structurally. A particular training run is an upfront expense carried by the model builder, while inference is a variable cost paid by whoever runs the model every time it is used. For a product team, that means inference is the line item that moves with every new user.
Three billing shapes
Inference is billed in three common shapes:
| Billing shape | Typical unit | Where you meet it | Hidden variable |
|---|---|---|---|
| Per token | Per million tokens, split into input and output | Hosted model APIs | Output length and context size |
| Per request, image or minute | Per call, per image, per minute of audio | Endpoints that resell model access | Request mix |
| GPU-hours or instance-hours | Hourly instance price | Self-hosted models | Utilization |
The first shape rewards efficient prompts. The third rewards efficient scheduling, because you buy hardware time whether or not it is busy.
How to calculate inference cost
A budget starts with a formula you can run against real traffic. Both hosted and self-hosted cases reduce to arithmetic.
Hosted API: cost per request and per month
For a token-priced API, cost per request equals (input tokens × input price + output tokens × output price) / 1,000,000 when prices are quoted per million tokens. Use separate rates for cache reads, cache writes and any other billed categories. Monthly cost is requests per month multiplied by that figure. A token is roughly four characters of English, according to the CloudZero cost calculator guide, which also states the monthly formula with separate input and output terms.
An illustrative worked example: a chat application receives 1 million requests per month, each with 800 input tokens and 200 output tokens. That is 800 million input tokens and 200 million output tokens. At rates of $2 per million input tokens and $10 per million output tokens, $3,600 per month is the resulting token cost before any cache, tool or regional surcharge. The arithmetic: 800 × $2 = $1,600 for input, 200 × $10 = $2,000 for output.
Notice that output is 20% of the tokens but about 56% of the token bill. That ratio is the first thing to examine in any cost review.
Why output tokens cost more
Output tokens often have a higher rate than input tokens, with the ratio depending on the model and provider, because each output token needs its own decode step, while input is processed in a single parallel pass. Meta's deployment guidance explains the same asymmetry through the autoregressive nature of LLMs and cites input/output token-count ratios of 4:1 or 3:1 as common rules of thumb when projecting costs.
Cached input is cheaper again. For example, Anthropic's official pricing lists model-dependent cache-read discounts and a 50% Batch API discount for eligible requests; cache writes have their own charges. Rates change often, so read the current provider page before you put numbers into a budget.
Self-hosted: cost per million tokens
For a model you host, the calculation inverts. For the GPU component, cost per million output tokens is hourly GPU cost divided by total output tokens produced per hour, multiplied by one million. If throughput is measured while busy, multiply aggregate output tokens per second by 3,600 and the busy-time fraction. If throughput is already averaged over wall-clock time, do not apply utilization again. Add hosting and operations costs for a full comparison. At the same busy-time throughput and hourly price, 25% busy time makes the GPU cost per output token four times that at 100%.
To size infrastructure, NVIDIA's engineers recommend you fix a latency target, such as a time-to-first-token requirement for chat, and a planned peak request rate. Total cost is then the number of servers times the yearly cost per server, which you can express as cost per 1,000 prompts or per million tokens.
What drives the bill
Several factors compound each other.
- Model size and tier. A frontier model and a small model from the same vendor can differ in price by more than an order of magnitude. A larger model is justified only on tasks where it measurably raises success.
- Context length. Autoregressive decoding uses the model weights and growing KV cache, and it can be memory-bandwidth-bound; the bottleneck depends on batching, model architecture and hardware. That is why longer conversations get slower and more expensive.
- Output length. Higher output rates make verbose responses and long reasoning traces expensive. Tool results generally add input tokens when fed back into context; their cost depends on length, reuse and provider pricing.
- Request volume and retries. Failed calls that are retried, and agents that loop, multiply token spend without producing value.
- Hardware and operations. For on-premises setups, Meta lists accelerator hardware, datacenter, operating staff and electricity as the main cost drivers.
Agent workloads sharpen all of these. An agent harness makes many model calls per task, carries context between them and invokes tools, so one user action can become dozens of requests. Our write-up on why Claude Code gets expensive covers how that multiplication arises and which changes reduce it.
Hosted API versus self-hosted inference
The choice is a question of utilization and scale. A pay-per-token hosted API charges for billed input and output rather than idle hardware; reserved capacity and other fees can change that comparison. A self-hosted deployment on rented or owned GPUs charges for time, so an idle accelerator is a fixed bill spread across few tokens.
| Factor | Hosted API | Self-hosted GPUs |
|---|---|---|
| Cost unit | Tokens | GPU-hours |
| Cost at low volume | Low, scales from zero | High, fixed floor |
| Cost at steady high volume | Linear with tokens | Can fall per token if utilization is high |
| Operations burden | Provider handles it | Your team handles scheduling, scaling, upgrades |
| Model choice | Provider catalog | Any open-weight model your hardware fits |
As a first approximation, divide the full monthly self-hosting cost by the effective API price per token for the same model quality and input/output mix, to estimate a break-even monthly token volume. Compare your expected volume with that point and check whether the hardware can serve it within its latency target. Below that point, pay-per-token hosting is cheaper under those assumptions. Well above it, self-hosting may win if the hardware can sustain that workload. Most teams find it practical to start on hosted models and revisit the decision when traffic is stable enough to forecast.
How to reduce inference costs without losing quality
Cost reduction is only real if outcomes hold steady. The measurement comes first, then the levers.
Measure cost per outcome
Cost per request can mislead. A 20% cut in cost per request that drops task success from 90% to 70% is a bad trade, because cost per successful outcome (cost per request divided by success rate) rises. Dan McCreary's token efficiency chapter makes this argument, and it is the right metric for any agent that completes tasks, since a cheaper model that needs too many retries may cost more per successful outcome.
The levers, in order of effort
- Cache stable context. System prompts, tool definitions and reference documents repeat across requests. Cache reads may cost a fraction of uncached input; check the provider's cache-read and cache-write rates and eligibility rules.
- Use batch modes for work that can wait. Evaluations, backfills and overnight jobs rarely need real-time answers, and batch pricing is commonly discounted by half.
- Cap output. Set output limits where supported, request structured responses and trim irrelevant tool results while preserving information needed for the task.
- Match the model to the task. Route simple classification and extraction to small models and reserve large models for hard reasoning. Our guide to LLM routers explains how model routing works, and the LLM proxy overview shows where a proxy stops and a fuller layer begins.
- Trim context. Summarize or drop old turns so the KV cache and input tokens do not grow without limit.
- Benchmark before switching. Compare candidates on your own prompts, scoring quality and cost together.
Model choice can be a major lever, but the savings depend on your workload. Compare current model rates using the same measured input/output mix and include any changes in retries and task success. The right model is the cheapest one that clears your quality bar on your own tasks.
Estimating and monitoring as usage changes
Budgets drift because three inputs change at once: traffic, model choice and prompt design. A durable process looks like this:
- Baseline. Record average input tokens, output tokens and request count per task type.
- Forecast. Apply the formulas above to peak and average traffic, with a margin for retries.
- Attribute. Tag spend by feature, tenant and model so you can see which workload moves the bill.
- Alert. Track cost per task and cost per outcome, and alert on step changes after a prompt or model change.
- Re-benchmark. When a provider releases a new tier or changes a rate, rerun your benchmark set rather than assuming the old ranking holds.
Rates vary by model, provider, token category and sometimes context length. Use the current provider rate card rather than treating a historical price range as a budget bound.
Where HarnessRouter fits
Inference cost control for agent products has an integration problem underneath it: each harness, from Codex to Claude Code to Hermes, has its own runtime, session handling and billing path. HarnessRouter is the world's first unified interface for agent harnesses. It exposes supported agent harnesses through one Agent API, with session management, file handling, plugin support, environment provisioning and benchmarking tools. Because the API is the same across supported harnesses, comparing compatible configurations on the same task typically requires configuration rather than a new backend integration.
Our positioning is that this comes without markup on model provider costs, and with both self-hosted and managed options. The Community Edition is the self-hosted route.
The cloud plans are published on the pricing page:
| Plan or fee | Price |
|---|---|
| Free | $0 per month |
| Developer | $20 per month |
| Production | $200 per month |
| Scale | $1,000 per month |
| Tracing | Included with Scale during beta; $50 monthly add-on for other plans |
| BYOK routing fee | No fee up to $25,000 monthly model usage at listed prices; above that volume, a separate 5% fee |
| Top-up credit processing fee | 5.5%, minimum $0.80 |
Enterprise pricing is available on request; agent work has published plan-dependent rates per active work minute. Teams that bring their own provider keys can do so with the BYOK option, and per-model rates are listed in the model pricing catalog.
To compare harness choices before you build, start from the Codex versus Claude Code comparison, then run your own task set through the Agent API.
If you are ready to benchmark harnesses and models on your own workload, the HarnessRouter pricing page shows which plan matches your stage, starting with a Free plan at $0 per month.
Our perspective on inference cost
Most inference cost conversations start with the price per million tokens, and that is the wrong place to start. The price is an input to the real figure, which is cost per successful task, and that figure depends on the harness, the model, the context strategy and the tools wrapped around them.
In agent products the variance between configurations is wide. The same task can run through different harnesses and models with very different token consumption and success rates. A team that integrates one harness directly tends to stay with it because switching means rewriting the backend. We built HarnessRouter to reduce the integration cost of switching: one Agent API, many supported harnesses, and benchmarking tools to compare quality against price on your own tasks.
That changes the economics. When comparison is cheap, you choose the harness and model on evidence and you can revisit the choice whenever a new model or rate card appears. Open standards such as the Unified Harness Protocol, and an open-source edition you can self-host, keep that choice yours instead of locking it to one vendor.
Our advice to teams: benchmark first, set the budget second, and track cost per outcome from day one.
HarnessRouter Editorial Team
FAQ
What is the difference between AI inference costs and LLM inference costs?
AI inference cost covers running any trained model, including vision, speech and classification models. LLM inference cost is the language-model subset, usually billed in tokens. Both include compute, memory, networking and platform overhead, and both grow with usage.
How do I estimate my monthly LLM API bill?
Multiply monthly requests by average input tokens and the input rate, then add monthly requests times average output tokens and the output rate, with rates per million tokens. Add a margin for retries, cache misses and tool calls. Replace estimates with measured averages once you have a week of real traffic.
Why are output tokens more expensive than input tokens?
Input is processed in one parallel pass, while each output token needs its own sequential decode step that reads the model weights and the KV cache from memory. Output rates vary by model and provider; use the current rate card.
When does self-hosting beat a hosted API?
Self-hosting may cost less when full monthly hosting costs, quality, input/output mix and feasible throughput beat the pay-per-token alternative at your expected volume. At low or spiky volume, paying per token is usually cheaper because idle time costs nothing.
What is the fastest way to cut inference costs?
Start with prompt caching, batch processing for non-urgent jobs and shorter outputs, then match each task to the smallest model that meets your quality bar. Track cost per successful outcome so savings do not hide quality losses.
Sources
- What is AI Inference Cost?
- What controls the cost of AI inference in production?
- What is inference cost in AI?
- AI Inference Economics: A Framework for Investors
- LLM cost calculator: estimate token and inference costs
- Cost projection, Llama deployment guides
- Anthropic official pricing
- LLM Inference Benchmarking: How Much Does Your LLM Inference Cost?
- Pricing, Economics, and Async API Modes