Rate Limiter System Design: Algorithms and Architecture Diagram

Every public API needs a rate limiter, and most teams discover it the same way: one client with a retry loop, or one scraper, takes the service down for everyone else. A limiter caps how much any single key, user or IP can ask for, so one bad actor costs you a 429 instead of an outage.

The design questions are always the same. Which algorithm, where the limiter sits, where the counters live, and what happens when the counter store is down. This post answers each one and ends with an architecture diagram of a distributed limiter in front of an API.

What the limiter must do

• Decide in a few milliseconds whether a request is allowed, on every request.
• Apply different rules to different callers: per API key, per user, per IP, per route.
• Behave the same no matter which server instance handles the request.
• Tell clients clearly when they are limited and when to try again.
• Keep working, or fail in a predictable way, when its own storage has a problem.

The third point is what makes it a distributed systems problem. A limit of 100 requests per minute means nothing if each of ten gateway instances counts its own 100.

Five algorithms

The fixed window problem is easy to see: a client sends 100 requests at 12:00:59 and another 100 at 12:01:00, and both windows accept them. That is 200 requests in two seconds against a limit of 100 per minute.

Token bucket is the usual default for APIs because bursts are normal client behavior. A limit of "100 requests per minute, burst 20" maps directly to a refill rate and a bucket size. Stripe described running token bucket limiters on Redis in its post on scaling an API with rate limiters, alongside a separate limiter on concurrent requests.

Sliding window counters are the cheap way to remove the boundary problem when you prefer a hard per-window limit.

Where the limiter sits

You can enforce limits in three places, and many systems use more than one:

• At the edge (CDN or WAF). Coarse limits per IP to absorb floods before they reach you.
• At the API gateway. Per key and per route limits that need to know who the caller is. This is where most product limits live.
• Inside a service. Limits that depend on business data, such as "two concurrent exports per account".

A gateway can count locally (each instance has its own counters) or globally (all instances share a store). Local limits are free and good enough for coarse protection; divide the limit by the number of instances and accept the error. Product limits that customers pay for need a global store. Envoy, for example, supports both: a local rate limit filter and a global mode that calls an external rate limit service backed by Redis.

Counters in Redis

Redis is the common choice for global counters: it is fast, keeps counters in memory and expires keys on its own. The trap is the race between reading a counter and writing it back. Two gateways read 99, both allow the request, both write 100.

Keep each decision atomic. For a fixed window, INCR followed by EXPIRE on a new key is enough; the Redis INCR documentation shows rate limiting patterns built on it. For a token bucket, run a small Lua script that reads the bucket, refills it based on elapsed time, takes a token and writes it back in one step. Redis runs a script without interleaving other commands, so concurrent requests see a consistent bucket.

-- KEYS[1] = bucket key, ARGV = rate per second, capacity, now (ms), cost
local b = redis.call('HMGET', KEYS[1], 'tokens', 'ts')
local rate, cap, now, cost = tonumber(ARGV[1]), tonumber(ARGV[2]), tonumber(ARGV[3]), tonumber(ARGV[4])
local tokens = tonumber(b[1]) or cap
local ts = tonumber(b[2]) or now
tokens = math.min(cap, tokens + (now - ts) / 1000 * rate)
local allowed = tokens >= cost
if allowed then tokens = tokens - cost end
redis.call('HSET', KEYS[1], 'tokens', tokens, 'ts', now)
redis.call('PEXPIRE', KEYS[1], math.ceil(cap / rate * 1000))
return allowed and 1 or 0

Pass the time in from the caller or use the Redis TIME command; mixing clocks from many gateways is a common source of odd results. If the same Redis also serves as a cache, watch its eviction policy: an evicted bucket is a reset limit. The caching strategies post covers eviction settings.

Telling the client

Answer a limited request with 429 Too Many Requests, defined in RFC 6585, and include a Retry-After header with the number of seconds to wait. Good clients back off; the others at least get a clear signal. An IETF draft, RateLimit header fields for HTTP, proposes standard headers that tell clients their quota and remaining budget before they hit the limit; at the time of writing (October 2026) it is still a draft, so treat any header names from it as provisional.

Document the limits. A limit that clients discover by being blocked turns into a support ticket.

When Redis is down

Decide this before it happens:

• Fail open. Allow requests when the limiter cannot decide. The API stays up, and you lose protection for the duration. This is the usual choice for product limits.
• Fail closed. Reject requests. Right for expensive or abuse-prone operations such as login attempts or SMS sends.
• Fall back to local limits. Each gateway enforces a coarse local limit until the store returns.

Whatever you choose, alert on it. A limiter that silently fails open for a week is no limiter.

The architecture diagram

How to read it:

• Left to right is the request path: clients, the load balancer, the API gateway and the rate limit service.
• The rate limit service owns the rules and makes one decision per request. It keeps counters in Redis and updates them with an atomic Lua script.
• The decision node sends allowed requests to the Orders API. The rest get a 429 with Retry-After and never reach the backend.
• Metrics are dashed because they are reported asynchronously. Rejected counts per rule are the first thing to check when a customer says they are blocked.
• The Markdown block lists the rules in the order they are checked, including the failure mode.

What to put on the diagram

• The rules and their keys. "100/min" is meaningless without "per API key" or "per IP".
• Local or global counting. The difference decides whether limits hold across instances.
• The failure mode. Fail open or fail closed belongs on the diagram, not only in code.
• The rejection path. Show where a 429 is produced so nobody assumes the backend sees those requests.

For the gateway itself, the API gateway architecture diagram covers routing, authentication and the other filters that run next to the limiter. The system design diagram example shows how a limiter fits into a larger design.

Build it in Archboard

1. Sign in with Google or GitHub and create a diagram.
2. Add a Person from the C4 Model pack for the clients, then a Cloud LB and an API Gateway from the Cloud pack.
3. Add two Container nodes from the C4 Model pack: the rate limit service and the Orders API. Add Redis from the Data pack below the rate limit service.
4. Add a Decision and a Terminator from the Flowchart pack for the allow or reject branch, and Metrics from the DevOps pack.
5. Label the edges, draw the metrics edge dashed, and add a Markdown block from the Core pack with the rules.

The example uses 10 diagram nodes, within the Free plan.

FAQ

Which rate limiting algorithm should I use? Token bucket for most APIs, because it allows the short bursts real clients produce. Use a sliding window counter if you need a strict limit per window without the cost of a full request log.

Should the rate limiter be a separate service? Not necessarily. A gateway filter with a shared Redis is enough for many systems. A separate service helps when many gateways and services share the same rules.

What status code should a rate limited request get? 429 Too Many Requests, with a Retry-After header. Do not use 503; it tells clients your service is down rather than that they asked for too much.

How do I rate limit by user when requests come through many servers? Keep the counters in a shared store such as Redis, keyed by user or API key, and update them atomically. Local counters per server only work as a rough cap.

What happens if the rate limiter itself fails? Choose in advance. Fail open for normal product limits, fail closed for expensive or abuse-prone operations, and alert in both cases.

Start free — one seat, no credit card.