Charged twice

Each payment is one customer pressing “Pay” once. The client sends the request, waits up to a timeout and, hearing nothing, sends it again. A lost reply looks exactly like a lost request — the client cannot know whether the card was charged — so it retries either way.

Without idempotency keys the server cannot tell that retry from a new payment, and charges again. With 15% of replies lost, about 15% of payments end up charged twice; a test checks that, with nothing else going wrong, the two rates match. With keys on it is zero at any loss rate, though each lost reply still costs the customer a full timeout: p99 latency rises from about 0.55 s on a clean network to over 2 s.

The client generates one key per logical operation, typically a random UUID, and sends it with every retry; Stripe’s API, for one, takes it in an Idempotency-Key header. The server stores the key with a fingerprint of the request and its response, in the same transaction as the charge. A repeat of a completed key gets the stored response and charges nothing; a repeat while the first is still in flight gets 409 Conflict and retries later; the same key with a different body gets 422.

Keys expire after a retention window, and the window has to outlast the client’s retries: a retry that arrives once its key is forgotten is a new payment, and is charged. Stripe keeps a key for at least a day. Here a payment sends its last retry within 74 seconds of its first attempt at the most, but a server that cannot keep up may get to it many minutes later: with keys kept for only a minute, that server charged twice with idempotency on.

Keys make a retry safe, but it still lands on the server. In the storm preset, fixed 100 ms retries push the busiest second after the outage to about 31 requests, five times the arrival rate; exponential backoff with full jitter, from AWS’s “Exponential Backoff and Jitter”, holds it to about 25 (means of 40 seeded runs). The backoff does that work: without jitter the peak is about 24, as jitter halves the average wait. Jitter earns its place elsewhere — clients that failed together, like everyone the crash reset, stop retrying together: in a test it spreads their fifth attempts four times wider.

At Worldline I handled the edge cases of the critical transaction path in 3-D Secure, integrated with acquirers and issuers. A request that arrives twice is the classic one: repeating it is safe only if the server can recognise it, which is what an idempotency key is for.

The same holds for events: at-least-once delivery, which the outbox experiment gives, is only safe when the receiving side is idempotent.