Idempotency: designing operations that survive retries
A network call can fail after the server has done the work. The client does not know, so it retries. Idempotency is what makes that retry safe, and it has to be designed in, not added later.
Originally written on 15 April 2019. Migrated from a WordPress blog and reformatted.
An operation is idempotent if performing it more than once has the same effect as performing it once. In a single process this is a curiosity. Across a network it is a requirement, because every remote call has a failure mode in which the request was processed but the response was lost, and the only correct client behaviour in that case is to try again.
Where retries come from
- A client sends a request, the server processes it, and the connection drops before the response arrives. The client sees a timeout.
- A message consumer processes a message and crashes before acknowledging it. The broker redelivers.
- A load balancer retries a request that took too long, while the original is still running.
- A user clicks a button twice.
In each case the operation runs twice. If it is "set status to SHIPPED", nothing is wrong. If it is "charge the card", the customer is charged twice.
Naturally idempotent operations
Some operations are idempotent by construction and need no further work.
| Operation | Idempotent | Reason |
|---|---|---|
UPDATE orders SET status='SHIPPED' WHERE id=42 | yes | absolute value |
UPDATE accounts SET balance = balance - 10 | no | relative change |
PUT /users/42 with a full body | yes | replaces state |
POST /orders | no | creates a new resource each time |
DELETE /orders/42 | yes | second call is a no-op (or 404) |
INSERT INTO log VALUES (...) | no | appends a row each time |
The pattern: absolute writes are idempotent, relative writes and creations are not. Where the domain allows, prefer absolute writes. "Set balance to 90" instead of "subtract 10" requires the caller to know the current balance, which is often a feature.
Making non-idempotent operations safe
For creations and relative writes, attach a unique key to the request and have the server remember it.
Idempotency key. The client generates a UUID per logical operation and sends it with every attempt. The server stores the key with the result of the first execution and returns the stored result for any repeat.
-- inside the same transaction as the work
INSERT INTO idempotency_keys (key, response, created_at)
VALUES ($1, $2, now())
ON CONFLICT (key) DO NOTHING;If the insert affects zero rows, the key was seen before: return the stored response without doing the work. Payment providers (Stripe's Idempotency-Key header is the well-known example) implement exactly this.
Two details that matter. The key store write must be in the same transaction as the side effect, or a crash between them re-opens the window. And keys need a retention period longer than any plausible retry, typically 24 hours or more.
Unique constraints as the key. When the operation has a natural identifier, use it. An invoice generated from event evt_123 can carry event_id as a unique column:
INSERT INTO invoices (event_id, amount) VALUES ($1, $2)
ON CONFLICT (event_id) DO NOTHING;No separate key table, and the database enforces it.
Concurrent duplicates
Two copies of the same request can arrive at the same time, not one after the other. The key store handles this if the insert happens first: the second transaction blocks on the unique constraint until the first commits, then sees the conflict. Checking for the key with a SELECT before doing the work does not handle it; both requests see no key and both proceed.
Ordering
Idempotency protects against the same operation twice. It does not protect against two different operations arriving out of order, such as shipped after delivered on a retry. That requires a version or timestamp on the entity and a conditional write:
UPDATE orders SET status = $2, version = $3
WHERE id = $1 AND version < $3;Summary
Assume every operation will be attempted more than once. Prefer absolute writes. For creations and relative changes, require a unique key, store it in the same transaction as the work, and rely on the database's unique constraint rather than a check-then-act sequence.