Two Copies of One Request Arriving at Once

A retry that arrives while the first request is still running finds no stored result and runs the payment again.

Idempotency keys fail when the server checks for a stored result and only records the key after acting. A retry that arrives while the first request is still running finds nothing stored and acts too. The key must be recorded atomically, as in progress, before any work starts, and a second copy that finds it in progress is answered with 409.

The symptom. Duplicate charges in the logs despite every request carrying a key, clustered around slow processor responses and aggressive client timeouts.

Two copies of one keyed payment, checked then recorded, both charged

Checking then recording, against recording atomically first

Check, act, then storea race window

Both copies look up the key before either has stored anything, so both find it missing and both charge. The window is as long as the charge takes.

Insert the key firstone winner

Each copy tries to insert the key as in progress with a unique constraint. One insert succeeds and charges; the other fails, and that copy is answered 409 and retries later to receive the stored result.

Why it happens. "Look up, and if missing, act" reads as one step but is two, and another request can run between them. It passes every test that sends one request at a time.

What it costs. The exact double charge the key was added to prevent, now with logs suggesting the protection works.

What to do. Make the first action an atomic insert of the key: a unique constraint in a database, or a conditional write such as DynamoDB's attribute_not_exists. Only the request that wins the insert does the work. Test it by sending two copies at the same moment.

← → move through sections and cards