Add a token-bucket rate limiter to the public API
lumen-labs/gateway#207·by mateo-r·+94 −3 · 3 files·Python 3.12 · asyncio service
Precomputed example. This review was run once on a real pull request and stored. Nothing on this page talks to GitHub or Claude: no review runs, and nothing posts.
What the agent said
2 things to fix before merge
- The bucket never refills: `last` is overwritten before the elapsed time is computed.
- No lock around the bucket update: check whether this can interleave.
- A 429 without `Retry-After` leaves clients guessing.
Full summary and the reviewer's notes
Adds a per-key token bucket and wires it in front of every public route as Starlette middleware, keyed on the API key with the client IP as a fallback. The structure is clean and the tests cover the burst and the key isolation. One real defect: the refill math reads the timestamp it has just overwritten, so a bucket never refills, after capacity requests a key is locked out until the process restarts. Neither test can see it because neither advances the clock.
What this PR does. Every request to the public API now passes through a rate limiter. Each API key gets a bucket of `rate_burst` tokens; a request takes one token, and tokens come back at `rate_per_second`. When the bucket is empty the request is answered with 429 instead of reaching the route.
What was checked, and what was dropped. Checked: the middleware ordering (the limiter runs before request-id assignment, which is fine, a 429 does not need an id); the fallback to client IP for unauthenticated calls; `retry_after()` is defined but nothing calls it yet. Dropped: a style note about the dataclass field order; a question about `settings.rate_burst` defaults (they are in settings.py, outside this diff, and read sensibly).
-
Finding 1 of 3 Should fix gateway/limits.py:22
The bucket never refills: `last` is overwritten before the elapsed time is read
Why it mattersAfter `capacity` requests a key is rate-limited forever (until the process restarts). Every steady client hits a wall a few seconds after deploy.
gateway/limits.py @@ -0,0 +1,38 @@ import time from dataclasses import dataclass @dataclass class Bucket: tokens: float last: float class TokenBucket: """Per-key token bucket: `capacity` burst, refilled at `rate` tokens per second.""" def __init__(self, capacity: int, rate: float) -> None: self.capacity = capacity self.rate = rate self._buckets: dict[str, Bucket] = {} def take(self, key: str) -> bool: b = self._buckets.setdefault(key, Bucket(self.capacity, time.monotonic())) now = time.monotonic() b.last = now b.tokens = min(self.capacity, b.tokens + (now - b.last) * self.rate) if b.tokens < 1: return False b.tokens -= 1 return True def retry_after(self, key: str) -> float: b = self._buckets.get(key) if b is None or b.tokens >= 1: return 0.0 return (1 - b.tokens) / self.rateThe comment as drafted
Line 22 sets
b.last = now, then line 23 computes the refill as(now - b.last) * self.rate, which is now always0. So tokens are only ever consumed, never restored, and once a key spends its burst it is locked out for the life of the process. Compute the elapsed time first, then stamplast. The existing tests cannot catch this because neither one advances the clock; a test with a fakemonotonicwould.Suggested change: the author could apply this in one click on GitHubnow = time.monotonic() elapsed = now - b.last b.last = now b.tokens = min(self.capacity, b.tokens + elapsed * self.rate)Evidence and confidence
High confidence Lands inline on the diff
How to verify. Build `TokenBucket(capacity=1, rate=100.0)`, call `take("k")` twice with `time.sleep(0.1)` between them: the second call returns False and should return True.
Kept by the reviewer
Reproduced in a REPL in under a minute: the second `take()` after a sleep returns False. The suggested reorder is exactly right and the note about the tests not advancing the clock is the useful part. Posted as written.
-
Finding 2 of 3 Should fix gateway/limits.py:24
Check-then-decrement is not atomic: two concurrent requests can both pass with one token left
Why it mattersUnder load the limiter admits more requests than the configured burst, so the protection is weaker exactly when it is needed.
gateway/limits.py @@ -0,0 +1,38 @@ import time from dataclasses import dataclass @dataclass class Bucket: tokens: float last: float class TokenBucket: """Per-key token bucket: `capacity` burst, refilled at `rate` tokens per second.""" def __init__(self, capacity: int, rate: float) -> None: self.capacity = capacity self.rate = rate self._buckets: dict[str, Bucket] = {} def take(self, key: str) -> bool: b = self._buckets.setdefault(key, Bucket(self.capacity, time.monotonic())) now = time.monotonic() b.last = now b.tokens = min(self.capacity, b.tokens + (now - b.last) * self.rate) if b.tokens < 1: return False b.tokens -= 1 return True def retry_after(self, key: str) -> float: b = self._buckets.get(key) if b is None or b.tokens >= 1: return 0.0 return (1 - b.tokens) / self.rateThe comment as drafted
take()readsb.tokens, compares it to 1, and then decrements: three steps with no lock. Two requests arriving together can both observetokens == 1, both pass the check, and the bucket goes to-1. Wrap the read-check-decrement in athreading.Lock(or anasyncio.Lockper key) so the admission decision is atomic.Evidence and confidence
Medium confidence Lands inline on the diff
How to verify. Fire 50 concurrent requests at a key with capacity 1 and count the 200s.
Dropped by the reviewer
The reasoning would hold for a threaded server, but this gateway is asyncio on a single event loop and `take()` contains no `await`, so it runs to completion before any other request can touch the dict. A lock here would be dead weight. A convincing finding, which is why a person has to check it.
-
Finding 3 of 3 Nit gateway/middleware.py:13
429 without a Retry-After header
Why it mattersWell-behaved clients cannot tell how long to back off, so they either hammer the endpoint or wait far longer than needed.
gateway/middleware.py @@ -1,8 +1,22 @@ from starlette.middleware.base import BaseHTTPMiddleware from starlette.responses import JSONResponse from .limits import TokenBucket from .settings import settings limiter = TokenBucket(capacity=settings.rate_burst, rate=settings.rate_per_second) class RateLimitMiddleware(BaseHTTPMiddleware): async def dispatch(self, request, call_next): key = request.headers.get("x-api-key", request.client.host) if not limiter.take(key): return JSONResponse({"error": "rate limited"}, status_code=429) return await call_next(request) class RequestIdMiddleware(BaseHTTPMiddleware): async def dispatch(self, request, call_next):The comment as drafted
A 429 with no
Retry-After. Sloppy: clients have no idea when to come back, and you already wroteretry_after()so there is no excuse for not using it.Evidence and confidence
High confidence Lands inline on the diff
How to verify. Exhaust a key and inspect the 429 response headers.
Edited by the reviewer
Right call, wrong tone: "sloppy" and "no excuse" are not a review, they are a mood. Kept the fact that `retry_after()` already exists and turned it into the concrete suggestion.
Posted instead
Could the 429 carry a `Retry-After` header? `limiter.retry_after(key)` already computes the value, so it is one line: `JSONResponse(..., headers={"Retry-After": str(math.ceil(limiter.retry_after(key)))})`. Clients that honour it stop hammering the endpoint while they are limited.
The review that posts: 2 of 3 comments, as one comment review, under the reviewer's own name
-
Should fixgateway/limits.py:22inline
Line 22 sets
b.last = now, then line 23 computes the refill as(now - b.last) * self.rate, which is now always0. So tokens are only ever consumed, never restored, and once a key spends its burst it is locked out for the life of the process. Compute the elapsed time first, then stamplast. The existing tests cannot catch this because neither one advances the clock; a test with a fakemonotonicwould.now = time.monotonic() elapsed = now - b.last b.last = now b.tokens = min(self.capacity, b.tokens + elapsed * self.rate) -
Nitgateway/middleware.py:13inline· edited
Could the 429 carry a
Retry-Afterheader?limiter.retry_after(key)already computes the value, so it is one line:JSONResponse(..., headers={"Retry-After": str(math.ceil(limiter.retry_after(key)))}). Clients that honour it stop hammering the endpoint while they are limited.
Nothing was sent from this page: the review above is what the reviewer's click would post from their own ReviewStage, with their own GitHub account.
Run this on your PR
One command on your laptop. Your GitHub account, your Claude subscription, your call on every comment: nothing posts until you click.