LLM Gateway Architecture: Routing, Caching, Fallbacks and Cost Control

A dozen features in your company call three model providers. Each team keeps its own API key, nobody knows which team spent what last month, and when the main provider returned 429 errors for an hour, every app failed differently. One switched models and broke its JSON output.

An LLM gateway, also called an AI gateway or LLM proxy, puts every model call through one service that owns the keys, the routing and the meter. This post walks the request path stage by stage, then covers routing, caching, token limits, metrics, security and the gateways you can adopt.

Why put a gateway in front of model providers

Provider limits and invoices do not follow your org chart. OpenAI sets rate limits per organization and project, not per user, and Anthropic per organization with optional workspace limits. A nightly batch job in the same project as a customer chat can use up the chat's tokens per minute. A gateway gives you one place for keys, routing, limits per team and cost attribution.

It does not replace your API gateway. An API gateway counts requests; an LLM gateway counts tokens and dollars, understands streamed responses and treats a provider's 429 as a reason to try another model.

The request path, stage by stage

Cheap checks run first: nothing reaches a provider until the request is allowed and not already answered.

1. Authenticate. Map the virtual key to an app, a team and its allowed models, or return 401.
2. Check limits. Compare the team's token counters and budget in Redis. Over the limit, return 429 with Retry-After.
3. Apply policy. Enforce the model allowlist and a max_tokens cap, and mask personal data such as email addresses.
4. Check the cache. A hit returns the stored answer with no provider call.
5. Route. Pick a model and an ordered fallback chain.
6. Call the adapter. Translate the request to the provider's API, attach the provider key and start a first-token timeout.
7. Retry or fall back on 429, 5xx, Anthropic's 529 overload error or a timeout, but only before the first token reaches the app. A provider can send an error after a 200 has started the stream, and switching models mid-answer stitches two replies together.
8. Stream and meter. Pass server-sent events through and write a usage event asynchronously. OpenAI's Chat Completions API sends usage in a final chunk only with stream_options.include_usage, and a cut stream loses it, so keep reading after a client disconnects.

Steps 1 to 4 should be memory or Redis lookups; logging and evaluations stay off the request path.

Routing strategies

The safest fallback is the same model on another host, such as a Claude model on the Anthropic API and on Amazon Bedrock. A different model family needs its own evals, since tool calls and JSON output break first, and every fallback needs spare quota.

Retry in one layer only: the OpenAI Python SDK retries twice by default, and LiteLLM's docs warn that stacked retries turn one request into (1 + N)² upstream calls. Rest a failing deployment instead of hammering it, as LiteLLM does for 5 seconds after a 429 by default; the circuit breaker pattern generalizes this.

Caching: exact, semantic and provider-side

Exact-match caching is safe but narrow: Cloudflare AI Gateway hashes the provider, endpoint, model, auth header and full body, so any change is a new entry. Put the tenant in your cache key, or one customer's answer can reach another. Semantic caching also matches paraphrases, but "cancel my order" and "cancel my subscription" can land close together, so use it only where a near miss is harmless.

Provider caching often saves the most. Anthropic caches prefixes marked with cache_control for 5 minutes, refreshed on each hit, and cache reads cost 0.1 times the base input price on most models. OpenAI caches by default on supported models, with cached input up to 95% cheaper (at the time of writing, November 2026). Keep system prompts byte-identical, with no timestamps at the top, redact deterministically, and pin a conversation to one deployment, because each provider account has its own cache.

At an assumed $3 per million input tokens, a 6,000-token system prompt plus a 500-token question costs $0.0195. With the prefix read from cache, 6,000 × $0.30/M + 500 × $3/M = $0.0033, about 83% less per hit. The caching strategies post covers invalidation.

Token rate limits and team budgets

Requests per minute are a poor limit for LLM traffic: one request can carry 50 tokens or 200,000. Limit tokens per minute per team, cap max_tokens and set a monthly budget in dollars. A request's size is unknown until it finishes, so pick an approach:

• Reserve, then reconcile. Count estimated input plus max_tokens up front, then correct. OpenAI counts the larger of max_tokens and a character-based estimate against its own limits.
• Count as you go. Anthropic counts output tokens as they are generated, and cache reads do not count toward input tokens per minute on most models.
• Count afterwards. Simple, but bursts overshoot. Cloudflare says concurrent requests can briefly exceed its spend limits.

Keep counters in Redis, shared by all gateway instances; the rate limiter system design covers the algorithms and a Redis outage. Price usage with a versioned price table, alert at 80% of a budget, and decide whether 100% means a hard stop or a cheaper model.

What to measure

Track time to first token per provider and model, input, output and cached tokens per team, cost per team and feature, errors by type, and fallback and cache hit rates. OpenTelemetry's generative AI conventions name attributes such as gen_ai.usage.input_tokens but are still in development, so pin the version you emit. The observability architecture diagram shows where the signals go.

Security: what a gateway does not solve

A gateway holds every provider key and sees every prompt, so give it few admins, audited config changes and logs that keep metadata for every call but prompts only sampled, redacted and short-lived.

It helps against what the OWASP Top 10 for LLM applications calls unbounded consumption: limits and budgets stop a looping agent or a leaked key from running up the bill. It does not solve prompt injection; OWASP says it is unclear whether fool-proof prevention exists, and instructions hidden in a retrieved document look like data to a gateway. Defenses are least-privilege tools, approvals and strict output handling, as in the AI agent architecture diagram.

Build or adopt

Envoy AI Gateway joined the Agentic AI Foundation as Agent Router in September 2026. Build only what is specific to you, such as routing by data class; the expensive part is keeping provider adapters current. A managed gateway puts a third party in every prompt's path; a self-hosted one makes Redis, Postgres and upgrades your job.

The LLM gateway diagram

How to read it:

• Two apps call one OpenAI-format endpoint with their own virtual keys and no provider key.
• Five stages run in order, with a note above each on what it decides. A request can stop at Auth (401, 429) or at a cache hit.
• Three targets: OpenAI as primary, Anthropic as dashed fallback on 429 or 5xx, and a self-hosted vLLM model for sensitive data.
• Below the stages: Postgres for keys and budgets, Redis for counters and cached answers, and the usage log, written asynchronously.
• The Markdown block lists example settings for one team.

Drawing LLM gateway diagrams clearly

• Draw stages, not one box. A single gateway rectangle hides whether redaction happens before the cache.
• Put conditions on edges: "fallback on 429/5xx", "sensitive data", "miss".
• Mark where data leaves your network, so external providers look different from the self-hosted model.
• Dash the async paths, such as metering and logging.
• Leave the model catalog out; it belongs in config.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Frame from the Core pack named LLM gateway.
3. Inside it, add five Component nodes from the C4 Model pack in a row: 1 Auth, 2 Policy, 3 Cache, 4 Router and 5 Adapters, with a Core Text note above each.
4. Below them, add Postgres and Redis from the Data pack and a Data Warehouse for the usage and cost log.
5. Left of the frame, add two Container nodes from the C4 Model pack for the apps; on the right, two External System nodes for OpenAI API and Anthropic API and an Endpoint from the ML pack for the self-hosted model.
6. Draw labeled edges; dash the fallback and the usage events.
7. Add a Markdown block from the Core pack with the example settings.

If you keep the path as a Mermaid flowchart, the Mermaid converter turns it into an editable first draft. The example uses 15 diagram nodes, within the Free plan.

FAQ

What is an LLM gateway? A service that every model call passes through. It holds provider keys, routes and falls back between models, caches, enforces token limits and budgets, and records usage per team.

What is the difference between an AI gateway and an API gateway? An API gateway limits HTTP requests. An AI gateway also understands tokens, streaming and provider errors, so it can limit by tokens, attribute cost and fall back to another model.

What is the difference between semantic caching and prompt caching? Semantic caching runs in your gateway and returns a stored answer for a similar question. Prompt caching runs at the provider and reuses a processed prompt prefix; the model still writes a new answer.

How do you rate limit LLM requests by tokens? Keep a token bucket per team in Redis, reserve estimated input plus max_tokens before each call, correct it afterwards, and return 429 with Retry-After at the limit.

Can an LLM gateway prevent prompt injection? No. It can filter obvious attacks and cap damage with budgets, but injected instructions in documents look like data. Defend in the application with least-privilege tools and approvals.

Start free — one seat, no credit card.