Is an LLM proxy the same as an LLM gateway?

Not exactly, though the terms overlap in vendor marketing. A proxy focuses on transport and provider normalization, while a gateway adds identity-aware policy such as per-team budgets, quotas, and audit logs. Many products combine both, so evaluate by the responsibilities you need rather than the label.

Can I self-host an LLM proxy?

Yes, and several open-source proxies exist for exactly this. Self-hosting keeps prompts and keys inside your own network, but you then operate the instances, upgrades, and failover. If you need agent sessions and environments as well, the HarnessRouter Community Edition is open source and can be self-hosted.

Does a proxy add latency?

Yes, since every request takes an extra network hop and some processing. The overhead is usually small relative to model inference time, but measure it with your real payloads and streaming enabled, and look at tail latency instead of the average.

What happens if the proxy goes down?

It becomes an outage for every application behind it unless you plan for that. Run multiple instances, keep a stateless data plane, and cache configuration on each node so traffic continues if the control plane is unavailable. Test the failure path before you depend on it.

Do AI agents work through a standard LLM proxy?

They can use it for model calls, but the proxy does not manage agent sessions, workspaces, files, or the harness itself. Teams building agent products typically add a layer that handles those concerns, such as an agent API in front of the harnesses.

LLM Proxy: What It Does, Where It Stops, and When You Need More

TL;DR

An LLM proxy is a service that sits between your application and one or more model providers and exposes a single endpoint, so the application sends one request format and the proxy handles provider-specific authentication, transport, and response normalization. For a plain chat-completion workload that is often enough. For products built around AI agents, which hold sessions, read and write files, and run tools in a sandbox, a proxy covers only the model call, and the rest of the runtime remains your responsibility. This guide explains the proxy, its neighbors (the gateway and the router), the controls worth having, and where an agent harness layer begins.

An LLM proxy standardizes access to multiple providers behind one endpoint and one API key; a router chooses which model serves a request, and a gateway enforces identity, budgets, and policy.
The useful proxy features are retries, fallback across providers, load balancing, rate limiting, key isolation, and per-request logging of tokens and cost.
Keep the data plane stateless and cache control-plane facts such as pricing and routing rules on each node, so a control-plane outage does not stop traffic.
A proxy proxies model calls. Agent products also need session state, file handling, environment provisioning, and a way to swap the agent harness itself, which is the layer HarnessRouter provides.

What an LLM proxy is

Picture one request leaving your application. Without a proxy, your code must know the base URL, authentication scheme, request schema, streaming format, and error codes of each provider it calls. With a proxy, the application talks to one endpoint, typically in an OpenAI-compatible request shape, and the proxy translates the call for whichever upstream provider serves it.

The practical consequences are concentrated in four areas:

Single integration surface. Adding or replacing a provider becomes a configuration change in the proxy rather than a code change in every service.
Credential isolation. Provider API keys live in the proxy. Applications receive a scoped key that can be rotated or revoked without touching the upstream credentials.
Uniform observability. Every call passes through one point, so latency, token counts, errors, and cost can be recorded in one schema.
Central policy. Rate limits, allow-lists of models, and budget ceilings can be enforced once instead of per service.

These are the same reasons teams searching for a way to use multiple LLM providers through one API usually give: the first provider is easy, and the second and third multiply the integration surface.

Proxy, gateway, and router: three responsibilities

The three terms are used loosely, and vendors apply them to overlapping products. A workable distinction is by responsibility. One practitioner breakdown describes the proxy as the transport layer, the router as the decision layer that picks the model or provider, and the gateway as the policy layer that adds identity, rate limits, budget enforcement, and audit trails. In practice they are layers of one stack rather than separate products.

Layer	Core question	Typical responsibilities	State required
Proxy	How does the request reach the provider?	Request translation, authentication, streaming, retries	Minimal
Router	Which model should handle this request?	Cost, latency, or quality-based selection, fallback order	Routing rules, health data
Gateway	Who is calling, and what are they allowed to do?	Per-team keys, budgets, quotas, audit logs, policy	Identities, ledgers

If you want the decision layer in depth, our guide to what an LLM router is and how model routing works covers selection policies. The rest of this article concentrates on the transport and policy responsibilities, and on what falls outside all three.