Add a token-bucket rate limiter to the public API

lumen-labs/gateway#207·by mateo-r·+94 −3 · 3 files·Python 3.12 · asyncio service

Precomputed example. This review was run once on a real pull request and stored. Nothing on this page talks to GitHub or Claude: no review runs, and nothing posts.

What the agent said

2 things to fix before merge

2 should fix1 nit
  • The bucket never refills: `last` is overwritten before the elapsed time is computed.
  • No lock around the bucket update: check whether this can interleave.
  • A 429 without `Retry-After` leaves clients guessing.
Full summary and the reviewer's notes

Adds a per-key token bucket and wires it in front of every public route as Starlette middleware, keyed on the API key with the client IP as a fallback. The structure is clean and the tests cover the burst and the key isolation. One real defect: the refill math reads the timestamp it has just overwritten, so a bucket never refills, after capacity requests a key is locked out until the process restarts. Neither test can see it because neither advances the clock.

What this PR does. Every request to the public API now passes through a rate limiter. Each API key gets a bucket of `rate_burst` tokens; a request takes one token, and tokens come back at `rate_per_second`. When the bucket is empty the request is answered with 429 instead of reaching the route.

What was checked, and what was dropped. Checked: the middleware ordering (the limiter runs before request-id assignment, which is fine, a 429 does not need an id); the fallback to client IP for unauthenticated calls; `retry_after()` is defined but nothing calls it yet. Dropped: a style note about the dataclass field order; a question about `settings.rate_burst` defaults (they are in settings.py, outside this diff, and read sensibly).

  1. Finding 1 of 3 Should fix gateway/limits.py:22

    The bucket never refills: `last` is overwritten before the elapsed time is read

    Why it mattersAfter `capacity` requests a key is rate-limited forever (until the process restarts). Every steady client hits a wall a few seconds after deploy.

    gateway/limits.py
    @@ -0,0 +1,38 @@
    import time
    from dataclasses import dataclass
    
    
    @dataclass
    class Bucket:
        tokens: float
        last: float
    
    
    class TokenBucket:
        """Per-key token bucket: `capacity` burst, refilled at `rate` tokens per second."""
    
        def __init__(self, capacity: int, rate: float) -> None:
            self.capacity = capacity
            self.rate = rate
            self._buckets: dict[str, Bucket] = {}
    
        def take(self, key: str) -> bool:
            b = self._buckets.setdefault(key, Bucket(self.capacity, time.monotonic()))
            now = time.monotonic()
            b.last = now
            b.tokens = min(self.capacity, b.tokens + (now - b.last) * self.rate)
            if b.tokens < 1:
                return False
            b.tokens -= 1
            return True
    
        def retry_after(self, key: str) -> float:
            b = self._buckets.get(key)
            if b is None or b.tokens >= 1:
                return 0.0
            return (1 - b.tokens) / self.rate
    

    The comment as drafted

    Line 22 sets b.last = now, then line 23 computes the refill as (now - b.last) * self.rate, which is now always 0. So tokens are only ever consumed, never restored, and once a key spends its burst it is locked out for the life of the process. Compute the elapsed time first, then stamp last. The existing tests cannot catch this because neither one advances the clock; a test with a fake monotonic would.

    Suggested change: the author could apply this in one click on GitHub
    now = time.monotonic()
    elapsed = now - b.last
    b.last = now
    b.tokens = min(self.capacity, b.tokens + elapsed * self.rate)

    Evidence and confidence

    High confidence Lands inline on the diff

    How to verify. Build `TokenBucket(capacity=1, rate=100.0)`, call `take("k")` twice with `time.sleep(0.1)` between them: the second call returns False and should return True.

    Kept by the reviewer

    Reproduced in a REPL in under a minute: the second `take()` after a sleep returns False. The suggested reorder is exactly right and the note about the tests not advancing the clock is the useful part. Posted as written.

  2. Finding 2 of 3 Should fix gateway/limits.py:24

    Check-then-decrement is not atomic: two concurrent requests can both pass with one token left

    Why it mattersUnder load the limiter admits more requests than the configured burst, so the protection is weaker exactly when it is needed.

    gateway/limits.py
    @@ -0,0 +1,38 @@
    import time
    from dataclasses import dataclass
    
    
    @dataclass
    class Bucket:
        tokens: float
        last: float
    
    
    class TokenBucket:
        """Per-key token bucket: `capacity` burst, refilled at `rate` tokens per second."""
    
        def __init__(self, capacity: int, rate: float) -> None:
            self.capacity = capacity
            self.rate = rate
            self._buckets: dict[str, Bucket] = {}
    
        def take(self, key: str) -> bool:
            b = self._buckets.setdefault(key, Bucket(self.capacity, time.monotonic()))
            now = time.monotonic()
            b.last = now
            b.tokens = min(self.capacity, b.tokens + (now - b.last) * self.rate)
            if b.tokens < 1:
                return False
            b.tokens -= 1
            return True
    
        def retry_after(self, key: str) -> float:
            b = self._buckets.get(key)
            if b is None or b.tokens >= 1:
                return 0.0
            return (1 - b.tokens) / self.rate
    

    The comment as drafted

    take() reads b.tokens, compares it to 1, and then decrements: three steps with no lock. Two requests arriving together can both observe tokens == 1, both pass the check, and the bucket goes to -1. Wrap the read-check-decrement in a threading.Lock (or an asyncio.Lock per key) so the admission decision is atomic.

    Evidence and confidence

    Medium confidence Lands inline on the diff

    How to verify. Fire 50 concurrent requests at a key with capacity 1 and count the 200s.

    Dropped by the reviewer

    The reasoning would hold for a threaded server, but this gateway is asyncio on a single event loop and `take()` contains no `await`, so it runs to completion before any other request can touch the dict. A lock here would be dead weight. A convincing finding, which is why a person has to check it.

  3. Finding 3 of 3 Nit gateway/middleware.py:13

    429 without a Retry-After header

    Why it mattersWell-behaved clients cannot tell how long to back off, so they either hammer the endpoint or wait far longer than needed.

    gateway/middleware.py
    @@ -1,8 +1,22 @@
    from starlette.middleware.base import BaseHTTPMiddleware
    from starlette.responses import JSONResponse
    
    from .limits import TokenBucket
    from .settings import settings
    
    limiter = TokenBucket(capacity=settings.rate_burst, rate=settings.rate_per_second)
    
    
    class RateLimitMiddleware(BaseHTTPMiddleware):
        async def dispatch(self, request, call_next):
            key = request.headers.get("x-api-key", request.client.host)
            if not limiter.take(key):
                return JSONResponse({"error": "rate limited"}, status_code=429)
            return await call_next(request)
    
    
    class RequestIdMiddleware(BaseHTTPMiddleware):
        async def dispatch(self, request, call_next):
    

    The comment as drafted

    A 429 with no Retry-After. Sloppy: clients have no idea when to come back, and you already wrote retry_after() so there is no excuse for not using it.

    Evidence and confidence

    High confidence Lands inline on the diff

    How to verify. Exhaust a key and inspect the 429 response headers.

    Edited by the reviewer

    Right call, wrong tone: "sloppy" and "no excuse" are not a review, they are a mood. Kept the fact that `retry_after()` already exists and turned it into the concrete suggestion.

    Posted instead

    Could the 429 carry a `Retry-After` header? `limiter.retry_after(key)` already computes the value, so it is one line: `JSONResponse(..., headers={"Retry-After": str(math.ceil(limiter.retry_after(key)))})`. Clients that honour it stop hammering the endpoint while they are limited.

The review that posts: 2 of 3 comments, as one comment review, under the reviewer's own name

  1. Should fixgateway/limits.py:22inline

    Line 22 sets b.last = now, then line 23 computes the refill as (now - b.last) * self.rate, which is now always 0. So tokens are only ever consumed, never restored, and once a key spends its burst it is locked out for the life of the process. Compute the elapsed time first, then stamp last. The existing tests cannot catch this because neither one advances the clock; a test with a fake monotonic would.

    now = time.monotonic()
    elapsed = now - b.last
    b.last = now
    b.tokens = min(self.capacity, b.tokens + elapsed * self.rate)
  2. Nitgateway/middleware.py:13inline· edited

    Could the 429 carry a Retry-After header? limiter.retry_after(key) already computes the value, so it is one line: JSONResponse(..., headers={"Retry-After": str(math.ceil(limiter.retry_after(key)))}). Clients that honour it stop hammering the endpoint while they are limited.

Nothing was sent from this page: the review above is what the reviewer's click would post from their own ReviewStage, with their own GitHub account.

Run this on your PR

One command on your laptop. Your GitHub account, your Claude subscription, your call on every comment: nothing posts until you click.

Install guide Try it live