Webhooks Missed While Your Endpoint Is Down

A webhook arrives on the sender's schedule. If your endpoint is down or slow when the sender calls, what happens next depends entirely on the sender: some retry for days, some never retry. A receiver that assumes either behaviour loses events or processes them twice.

The symptom. After a 40-minute outage, a shop finds paid orders still marked unpaid. A week later, after a short slowdown, other customers receive two confirmation emails for one order.

The trigger. The endpoint was unreachable during the outage, and slow during the slowdown, so senders counted those deliveries as failed.

The mechanism. The unpaid orders were GitHub-style losses: the sender tried once, and nothing tried again. The duplicate emails were Stripe-style retries: the first delivery was processed, its reply came too late, and the retry was processed as new.

The cost. Customers who paid and received nothing, customers sent two emails for one order, and hours spent matching records by hand.

The remedy. Do two things whatever the sender does. Record each event's id and ignore one already processed, so a retry changes nothing. And reconcile: after any outage, and on a schedule, ask the sender's API for recent events and process those you never saw. Reply 2xx as soon as the event is stored, so a slow job never turns a success into a retry.

So never rely on a webhook arriving exactly once: deduplicate by event id, and fetch what you missed.