Design a Rate Limiter System

A rate limiter controls the number of requests a client can make within a time window. I would design it as follows. First, requirements: support per-user and per-API rate limits, handle 10,000+ requests/second, work across a distributed server fleet, and add <5ms latency. I would choose the Token Bucket algorithm for its flexibility. Each user/API key has a bucket with a maximum capacity (burst size) and a refill rate (sustained rate). For example, capacity=10, refill=5/second means a burst of 10 requests, sustained rate of 5/second. When a request arrives, if the bucket has tokens, consume one and allow the request; otherwise, reject with HTTP 429. For distributed rate limiting, I would use Redis as the centralized token store. The check-and-update operation must be atomic to prevent race conditions. I would use a Redis Lua script: local tokens = tonumber(redis.call("get", key) or capacity); local last_refill = tonumber(redis.call("get", key..":time") or now); tokens = min(capacity, tokens + (now - last_refill) * refill_rate); if tokens >= 1 then tokens = tokens - 1; redis.call("setex", key, ttl, tokens); redis.call("setex", key..":time", ttl, now); return 1; else return 0. Lua scripts execute atomically in Redis, eliminating race conditions. Architecture: request → API gateway / middleware → Redis token bucket check → if allowed, forward to application server; if denied, return 429 with headers (X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, Retry-After). For fault tolerance, if Redis is unavailable, I would fail open (allow requests) with local in-memory rate limiting as a fallback — protecting users is better than blocking everyone. For multiple rate limit rules (100/minute AND 1000/hour), check all rules and allow only if all pass. At massive scale, use Redis Cluster with consistent hashing to distribute keys, and consider local counters with periodic sync for ultra-low latency.