Retry failed webhook deliveries with exponential backoff
northwind/dispatch#412·by priya-k·+61 −9 · 4 files·TypeScript · Node 22 service
Precomputed example. This review was run once on a real pull request and stored. Nothing on this page talks to GitHub or Claude: no review runs, and nothing posts.
What the agent said
2 things to fix before merge
- Retries 4xx responses that cannot succeed, each dead endpoint burns all five attempts.
- Backoff grows 2× per attempt; check the cap holds for a long outage.
- No jitter: simultaneous failures retry in lockstep.
Full summary and the reviewer's notes
Replaces the single-shot webhook POST with a bounded retry loop and a new backoffMs helper. The shape is right, a cap on attempts, a timeout per request, tests for the retry and the give-up path. One correctness problem: the loop retries on every non-2xx status, including the 4xx replies that can never succeed, so a receiver that returns 404 costs five attempts and 31 s of worker time per event. Two smaller notes on the backoff itself.
What this PR does. When an event happens, `deliver()` POSTs it to a customer's webhook URL. Before this PR a single failed request lost the event. Now the request is retried up to five times with a growing delay between attempts, and the error names how many attempts were made.
What was checked, and what was dropped. Checked: the loop terminates (attempt increments on every failure and the throw fires at MAX_ATTEMPTS); the per-request timeout aborts a hung receiver; the tests cover a transient failure and the give-up path. Dropped: a note about `sleep` being module-private (it is only used here); a question about whether `x-dispatch-attempt` should be 0- or 1-based (the header is new, either is fine).
-
Finding 1 of 3 Should fix src/webhooks/deliver.ts:25
Every non-2xx is retried, including 4xx replies that can never succeed
Why it mattersA receiver that answers 404 or 400 costs five requests and ~31 s of backoff per event; a disabled endpoint with a busy stream will hold the delivery worker for most of its time.
src/webhooks/deliver.ts @@ -14,9 +14,24 @@ import { backoffMs, MAX_ATTEMPTS } from "./backoff"; import { DeliveryError } from "./errors"; export async function deliver(hook: Webhook, payload: Payload): Promise<void> { const res = await fetch(hook.url, { method: "POST", body: JSON.stringify(payload) }); if (!res.ok) throw new DeliveryError(res.status); let attempt = 0; while (true) { const res = await fetch(hook.url, { method: "POST", headers: { "content-type": "application/json", "x-dispatch-attempt": String(attempt + 1) }, body: JSON.stringify(payload), signal: AbortSignal.timeout(10_000), }); if (res.ok) return; attempt += 1; if (attempt >= MAX_ATTEMPTS) throw new DeliveryError(res.status, attempt); await sleep(backoffMs(attempt)); } } const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));The comment as drafted
if (res.ok) return;treats every non-2xx the same, so a400,404or410from the receiver is retried five times with the full backoff even though it cannot succeed. Only408,425,429and5xxare worth retrying. Throw straight away on the other 4xx so a dead endpoint costs one request, not five and 31 seconds.Suggested change: the author could apply this in one click on GitHubif (res.ok) return; if (res.status >= 400 && res.status < 500 && ![408, 425, 429].includes(res.status)) { throw new DeliveryError(res.status, attempt + 1); }Evidence and confidence
High confidence Lands inline on the diff
How to verify. Point a hook at a URL that returns 404 and deliver one event: watch five POSTs land 31 s apart. With the fix, one.
Kept by the reviewer
Checked `deliver.ts:25`: `res.ok` is the only branch, so a 404 really does loop five times. The suggestion is a correct, minimal fix. Posted as written.
-
Finding 2 of 3 Should fix src/webhooks/deliver.ts:28
Backoff is unbounded, a long outage produces multi-minute sleeps
Why it mattersDuring a receiver outage the delay doubles every attempt with no ceiling, so late attempts sleep for minutes and the worker's queue backs up behind them.
src/webhooks/deliver.ts @@ -14,9 +14,24 @@ import { backoffMs, MAX_ATTEMPTS } from "./backoff"; import { DeliveryError } from "./errors"; export async function deliver(hook: Webhook, payload: Payload): Promise<void> { const res = await fetch(hook.url, { method: "POST", body: JSON.stringify(payload) }); if (!res.ok) throw new DeliveryError(res.status); let attempt = 0; while (true) { const res = await fetch(hook.url, { method: "POST", headers: { "content-type": "application/json", "x-dispatch-attempt": String(attempt + 1) }, body: JSON.stringify(payload), signal: AbortSignal.timeout(10_000), }); if (res.ok) return; attempt += 1; if (attempt >= MAX_ATTEMPTS) throw new DeliveryError(res.status, attempt); await sleep(backoffMs(attempt)); } } const sleep = (ms: number) => new Promise((r) => setTimeout(r, ms));The comment as drafted
backoffMs(attempt)doubles on every attempt (BASE_MS * 2 ** (attempt - 1)) and nothing caps it, so a sustained outage turns the fifth attempt into a very long sleep. Clamp the delay to a maximum (30 s is typical) so a down receiver degrades to a steady retry rate instead of ever-longer pauses.Evidence and confidence
Medium confidence Lands inline on the diff
How to verify. Fail every request and log `backoffMs(attempt)` for attempts 1 to 5.
Dropped by the reviewer
Wrong: `backoff.ts:9` clamps the delay with `Math.min(MAX_BACKOFF_MS, …)`. The reviewer read the call site in `deliver.ts` without opening the helper it calls. Plausible on its face, so it was worth checking rather than trusting.
-
Finding 3 of 3 Nit src/webhooks/backoff.ts:9
No jitter on the backoff
Why it mattersWhen a receiver goes down, every in-flight delivery fails together and then retries at exactly the same moments, hammering it in waves as it comes back.
src/webhooks/backoff.ts @@ -0,0 +1,12 @@ export const MAX_ATTEMPTS = 5; const BASE_MS = 1_000; const MAX_BACKOFF_MS = 30_000; /** * Delay before retry number `attempt` (1-based): 1 s, 2 s, 4 s, 8 s, 16 s, capped at 30 s. */ export function backoffMs(attempt: number): number { return Math.min(MAX_BACKOFF_MS, BASE_MS * 2 ** (attempt - 1)); }The comment as drafted
No jitter. This is the textbook mistake, every retry guide written in the last decade covers it. Add randomness to the delay or this will just DDoS the receiver in waves the moment it recovers.
Evidence and confidence
High confidence Lands inline on the diff
How to verify. Fail 50 deliveries at once and plot the retry timestamps: without jitter they land in five tight spikes.
Edited by the reviewer
The point stands, simultaneous failures retry in lockstep, but "textbook mistake" and "DDoS" are not how this team talks to each other. Kept the substance, said it plainly, and added the concrete range.
Posted instead
Consider adding jitter to the backoff (for example ±20% of the computed delay). When a receiver goes down, every in-flight delivery fails at the same moment and then retries at the same moments, so it comes back to a wave of requests. A little randomness spreads them out.
The review that posts: 2 of 3 comments, as one comment review, under the reviewer's own name
-
Should fixsrc/webhooks/deliver.ts:25inline
if (res.ok) return;treats every non-2xx the same, so a400,404or410from the receiver is retried five times with the full backoff even though it cannot succeed. Only408,425,429and5xxare worth retrying. Throw straight away on the other 4xx so a dead endpoint costs one request, not five and 31 seconds.if (res.ok) return; if (res.status >= 400 && res.status < 500 && ![408, 425, 429].includes(res.status)) { throw new DeliveryError(res.status, attempt + 1); } -
Nitsrc/webhooks/backoff.ts:9inline· edited
Consider adding jitter to the backoff (for example ±20% of the computed delay). When a receiver goes down, every in-flight delivery fails at the same moment and then retries at the same moments, so it comes back to a wave of requests. A little randomness spreads them out.
Nothing was sent from this page: the review above is what the reviewer's click would post from their own ReviewStage, with their own GitHub account.
Run this on your PR
One command on your laptop. Your GitHub account, your Claude subscription, your call on every comment: nothing posts until you click.