API rate limiting is the backstop between a working service and one that falls over when a client misbehaves, a batch job runs hot, or a scraper decides your endpoint is interesting. Most teams bolt it on after the first incident. That order of operations creates fragile defaults that survive light testing and crack under real traffic.
Getting rate limiting right means making design decisions early: which algorithm to use, where in the stack to enforce it, how to communicate limits to callers, and what happens at the boundary. None of those decisions are hard individually. The problems arrive when teams pick defaults without thinking through the failure modes.
The four main algorithms and when to reach for each
Token bucket is the most forgiving option. Each client has a bucket that fills at a fixed rate, and each request costs one token. Bursting is allowed up to the bucket's capacity. This works well for human-driven clients where occasional spikes are normal, such as a user dashboard that fires several requests when a page loads.
Fixed window counting is simpler to implement. Requests are tallied per time window (say, 100 requests per minute), and the count resets at the boundary. It's predictable but has an edge case: a client can exhaust its quota at the end of one window and immediately consume the next window's full quota, effectively doubling throughput at the seam. For low-risk internal services that edge case doesn't matter. For public APIs it does.
Sliding window log fixes that boundary problem by tracking each request's timestamp and counting only requests within the rolling window. It's accurate but memory-intensive. Store a log entry per request per client in Redis and you'll notice the memory cost when traffic scales.
Leaky bucket enforces a strict, constant output rate regardless of burst. Requests queue up; if the queue fills, they're dropped. This suits payment APIs or anything that feeds a downstream system that can't absorb spikes. It doesn't suit read-heavy APIs where brief bursts are harmless.
Where to enforce: gateway, middleware, or service layer
The gateway is the right place for most rate limiting. Products like Kong Gateway and AWS API Gateway apply limits before requests reach application code, which matters when the goal is protecting compute. A request that never reaches your service costs almost nothing. A request that runs a database query before being rejected costs real resources.
Middleware-level enforcement (a library inside the application framework) is common but has a drawback. By the time the middleware runs, the server has already accepted the connection and begun parsing the request. Under extreme load that's enough to tip a service over. Middleware limiting works best as a second layer, not the primary one.
Service-layer limiting, applied per endpoint inside individual microservices, makes sense when different endpoints have genuinely different cost profiles. A lightweight health check endpoint and a heavy report-generation endpoint shouldn't share the same limit. This is worth the complexity only if your microservices architecture is already mature enough to justify per-service configuration.
The distributed state problem
A single-server counter is trivial. Three load-balanced instances sharing state is not. If each instance counts requests independently, a client can hit all three and effectively triple its allowed rate.
The standard fix is a shared counter in Redis using atomic increment operations. Redis's INCR plus an expiry on the key is enough for fixed window limiting. Sliding window requires a sorted set. Both work at scale, but add a Redis dependency to your critical path. If Redis goes down, your rate limiter goes down. Most teams configure a fail-open policy (traffic passes through) rather than fail-closed (traffic is blocked), because blocking all traffic during a Redis outage is usually worse than briefly over-serving a client.
One underappreciated option is approximate limiting using local counters with periodic synchronisation. Each instance applies its own limit and syncs totals every few seconds. Clients can briefly exceed the global limit, but the overshoot is bounded. For most APIs that trade-off is acceptable and removes Redis from the critical path.
Communicating limits to callers
Rate limit headers are the cheapest improvement most APIs don't make. The RateLimit-Limit, RateLimit-Remaining, and RateLimit-Reset headers (as defined in the IETF draft RFC) tell a client exactly where it stands, which allows well-behaved clients to throttle themselves before hitting the wall.
When a client does hit the limit, return a 429 Too Many Requests response, not a 503. A 503 signals that your service is broken. A 429 signals that the client needs to back off. Include a Retry-After header with the number of seconds until the client's quota resets. Without that header, clients typically retry immediately and make the problem worse.
This connects to the broader point about API versioning: contracts with callers include behaviour under load, not just schema. If you change your rate limit policy without communicating it, you break clients that built their retry logic around the old numbers.
Common mistakes that bite later
Limiting by IP address alone is easy to implement and easy to bypass. A client behind NAT shares an IP with dozens of other users, and a single slow client can block all of them. Authenticated API key or user ID limits are more accurate and fairer.
Flat global limits ignore request cost. A 10 KB read and a 100 MB export are not the same request. Consider rate limiting on a cost unit rather than request count: compute-heavy or bandwidth-heavy operations can carry a higher token cost per call, which nudges clients toward efficient usage without a more complex billing model.
Not testing the rate limiter under real concurrency is the most common failure mode. Unit tests that fire sequential requests will pass. Load tests that send 500 concurrent requests from the same client often reveal race conditions in atomic operations or misconfigured key namespacing across instances.
Finally, treat rate limit configuration as code. Limits that live only in a gateway's web console drift over time and aren't version-controlled. Store them in infrastructure-as-code alongside the rest of the service configuration so changes are auditable and reproducible.