At-Least-Once Delivery Means Sometimes Twice

Most message queues promise at-least-once delivery: a message will be delivered, but possibly more than once. A worker that does its job and then fails before acknowledging the message will see it again, or another worker will, and without a check the job is done twice.

The symptom. A few customers each month are charged twice for one order. The two charges are minutes apart, and both came from the payment worker.

The trigger. A worker was restarted during a deploy, or ran out of memory, in the moment between charging the card and acknowledging the message.

The mechanism. The queue cannot know that the charge succeeded; it only knows that no acknowledgement arrived. From its side, a worker that crashed before charging and one that crashed after look the same, so it delivers again. That is exactly what keeps the first case from losing the order, and the price is a duplicate in the second.

The cost. Customers charged twice, refunds, and lost trust. With emails, a duplicate receipt; with stock counts, inventory reduced twice.

The remedy. Make doing the job twice harmless, which is called making it idempotent. Record each order id once it is charged and skip one already recorded, or pass an idempotency key to the payment provider, which then refuses to charge the same key twice. The subject on resilient systems covers idempotency in depth.

So assume every message will sometimes arrive twice, and make each consumer's work safe to repeat before it touches money or stock.