Card 07/ 09
Webhooks That Arrive Twice or Out of Order
A sender retries a delivery that succeeded, and "order paid" arrives after "order shipped".
Webhook senders promise at-least-once delivery, not exactly once and not in order. A delivery whose acknowledgement was lost is sent again, and two events sent close together may arrive the wrong way round when one is retried. A receiver must recognise a repeat by its event id and recognise an old event by its version or time.
The symptom. Customers receive two confirmation emails for one payment, and orders that have shipped flip back to paid in the receiver's system.
| Delivery | Event | Naive receiver's state | Checking receiver's state |
|---|---|---|---|
| 1 | evt_1 order.paid, version 2 | paid, email sent | paid, email sent |
| 2 | evt_1 order.paid again, after a lost 200 | paid, second email sent | paid; evt_1 already seen, skipped |
| 3 | evt_3 order.shipped, version 3 | shipped | shipped |
| 4 | evt_2 order.paid, version 2, a late retry | paid again | shipped; version 2 is older, ignored |
Why it happens. The sender cannot tell a receiver that processed an event and lost the 200 from one that never got it, so it resends, the problem from the idempotency keys topic in the other direction. Retries run on their own schedules, so a retried earlier event can overtake a later one.
What it costs. Repeated side effects, such as emails, charges and stock movements, and state that goes backwards.
What to do.
- Store each event id when it is processed, and skip any id already stored.
- Keep a version or timestamp on the receiver's copy, and ignore an event older than it.
- Or treat each event as a nudge and fetch the resource's current state.
- Answer 2xx as soon as the event is stored, and process it afterwards, so slow work does not cause retries.
← → move through sections and cards