Start here: the retry you can't avoid#
TL;DRthe 30-second version
- A network call can fail in a way that leaves you unsure whether it worked. A timeout looks the same whether the request never arrived or only the reply was lost. So the caller has to retry — which means the server will sometimes see the same request twice.
- Idempotency means applying the same request twice has the same effect as applying it once. A charge either happens or it doesn't, no matter how many copies arrive.
- The pattern: the client attaches an idempotency key — a unique id it picks for this one operation. The server keeps a dedup store (key → saved result). First time it sees a key, it does the work and saves the result. Every repeat of that key replays the saved result instead of redoing the work.
- You cannot buy 'exactly-once delivery' from the network — that's impossible. You build effectively-once processing: at-least-once delivery (keep retrying until one copy is confirmed) plus a receiver that ignores duplicates.
- The other half is retry discipline: exponential backoff plus jitter, so a wave of failures doesn't turn every client into a synchronized retry storm that keeps the server down.
Your service sends a request to charge a customer $50 for an order. It opens the connection, sends the charge, and waits. Nothing comes back. After a few seconds the call times out. Now you have to decide what to do — and you're missing the one fact you need: did the charge actually happen?
A timeout can't tell you. Maybe your request never reached the payment service, and no money moved. Maybe it reached the service, the charge succeeded, and only the reply got lost on the way back. From where you're standing, those two outcomes look exactly the same: you sent a request, you got silence. This is the core difficulty — a failure gives you no signal about how far the request got.
Build it: the idempotency key#
Start from the goal. You want the operation to be idempotent: applying the same request twice has the same effect as applying it once. Send the $50 charge one time or five times — the customer is charged $50, once. If you can guarantee that, retries become boring. You can resend as many copies as you like and never do harm.
So the server needs a way to tell 'this is a retry of a charge I already did' apart from 'this is a brand-new charge that happens to look similar.' That distinction is harder than it sounds. Two different customers can each buy a $50 item, one second apart — same amount, same product, genuinely two charges. You can't dedup on the contents of the request, because two legitimate requests can be identical.
The fix is to let the client name the operation. Before it sends the charge, the client generates an idempotency key: a unique id that stands for this one intended action. A common choice is a UUID (a random 128-bit id with essentially no chance of two clients picking the same one). The client attaches that key to the request, and — this is the important part — reuses the same key on every retry of that same charge. New charge, new key; retry of a charge, same key.
Now the server can dedup. It keeps a dedup store: a table that maps each idempotency key to the result of the request that carried it. The logic on every incoming request is three lines:
- Look up the request's idempotency key in the dedup store.
- If the key is new: do the work (charge the $50), then save (key → the response you produced) before replying.
- If the key is already there: skip the work entirely and reply with the saved response.
Walk the timeout through it. The first attempt arrives with key idem_9f8c, the server charges $50 and saves idem_9f8c → "charged, receipt #A1", but the reply is lost. The client times out and retries with the same key idem_9f8c. The server looks it up, finds it, and replays "charged, receipt #A1" without touching the customer's card. One charge, two requests. That's the whole pattern, and it's what Stripe exposes as the Idempotency-Key HTTP header: you put a key you chose in that header, and Stripe promises that two requests with the same key produce one result.
Exactly-once is impossible; effectively-once is the goal#
It's tempting to wish the problem away with a better network: what if delivery were just 'exactly-once', so every message arrived one time, guaranteed? Then you'd never need idempotency keys. The catch is that exactly-once delivery over an unreliable network is provably impossible, and it's worth knowing why, because it's a classic interview trap.
The sender has two honest options, and neither gives exactly-once. If it stops after one send and the acknowledgement is lost, the message might have been delivered zero times — that's at-most-once (you never double up, but you can lose messages). If it keeps resending until it gets an acknowledgement, the message is delivered one or more times, because an acknowledgement can always be the thing that gets lost — that's at-least-once (you never lose a message, but you can duplicate). There's no third setting on that dial that gives you precisely one delivery, because the sender can never be certain a delivery landed.
Since real systems can't afford to silently lose a payment, they choose at-least-once and live with duplicates. The trick is to move the 'once' from delivery to processing. Delivery can happen many times; processing happens once, because the receiver's dedup store throws away the duplicates. The industry name for the combination is effectively-once (or exactly-once processing): at-least-once delivery plus an idempotent receiver. 'Exactly-once' systems like Kafka's don't do magic on the wire — under the hood they do exactly this, retry plus dedup, and just hide it behind a clean API.
Go deeperUnder the hood: the Two Generals framing
The impossibility has a classic name: the Two Generals Problem. Two generals on opposite hills must agree on a time to attack, but the only way to communicate is a messenger who might be captured crossing the valley. General A sends 'attack at dawn.' Did it arrive? A needs an acknowledgement. But B's acknowledgement can also be captured, so B can't be sure A got it — so B wants an acknowledgement of the acknowledgement, and so on forever. No finite exchange of messages can make both sides certain.
A network call is exactly this: your request is one messenger, the reply is the acknowledgement, and either can be lost in the valley. That's why a timeout is ambiguous, and why no protocol can promise exactly-once delivery. The escape isn't to keep certainty on the wire — it's to make duplicate delivery harmless, which is precisely what idempotency does.
What it costs: the dedup window#
The dedup store can't grow forever. You keep a key around only long enough to catch the retries that matter, then delete it. That retention period is the dedup window — how long a key stays remembered. Stripe, for example, keeps idempotency keys for 24 hours; after that a repeat of an old key is treated as a brand-new request.
The window is a real trade-off. Too short, and a client that retries after the window has expired finds its key forgotten — the server treats the retry as new and does the work again (the double charge is back). Too long, and the dedup store swells with keys no one will ever resend. The rule of thumb: the window must be longer than the longest retry horizon any client will use. If a client might keep retrying a stuck request for an hour with backoff, a 24-hour window is comfortably safe.
PredictYour payments API takes 5 million charge attempts a day. You keep each idempotency key plus its saved response — about 400 bytes per entry — for a 24-hour window. Roughly how much does the dedup store hold? And what actually breaks if you cut the window to 5 minutes to save space?
Hint: Multiply attempts × bytes for the size. For the second half: what happens to a retry whose key was deleted before it arrives?
Size: a 24-hour window holds about one day of attempts, so 5,000,000 × 400 bytes ≈ 2,000,000,000 bytes ≈ 2 GB. That's small — dedup storage is cheap, which is why nobody shrinks the window to save space. What breaks at 5 minutes is correctness, not cost: any retry that arrives more than 5 minutes after the original finds its key already deleted, so the server treats the retry as a new charge and bills the customer again. Slow clients, a client that backs off for minutes during an outage, or a request stuck in a queue all blow past a 5-minute window. The window is sized by the client's maximum retry horizon, not by storage — and since storage is ~2 GB either way, you keep it generously long (hours to a day), never minutes.
There's a second cost, and it's the subtle one: two copies of the same key can arrive at the same time. The client's first attempt is slow, it times out and retries, and now the original and the retry are both in the server at once — a race. If both read the dedup store, both see 'key not found', and both charge the card. The dedup store only works if the first-writer-wins check is atomic.
The other half: retry without a storm#
Idempotency makes a retry safe. It does nothing about how often you retry — and getting that wrong turns a small outage into a big one. The failure has a name: the retry storm (or thundering herd).
Picture a server that slows down for a moment. A thousand clients' requests time out at nearly the same time. If every client retries immediately, the struggling server is hit by a thousand retries at once, on top of the normal load — so it slows down more, times out more requests, and triggers even more retries. The retries themselves become the outage. A server that would have recovered in a second stays down because the herd won't stop hammering it.
The first fix is exponential backoff: wait longer before each retry. Try after 1 second, then 2, then 4, then 8. That thins out one client's retries over time, so a client that can't get through backs off instead of spinning. But backoff alone isn't enough, because all thousand clients failed at the same moment — so they all wait 1 second, then all retry together, then all wait 2 seconds, then all retry together. The herd is still synchronized; you've just spaced out its stampedes.
The second fix breaks the synchronization: jitter — add a random amount to each wait. Instead of every client waiting exactly 2 seconds, each waits a random time between 0 and 2 seconds. Now the retries spread smoothly across the window instead of landing in a spike, and the recovering server sees a trickle it can serve rather than a wall it can't. Backoff plus jitter together is the standard retry policy, and it's what AWS's own SDKs ship with. AWS's canonical write-up on this concludes that jitter — the randomness, not just the backoff — is what actually decorrelates the herd.
When you don't need an idempotency key
Not every operation needs the full key-plus-dedup-store machinery. Some are naturally idempotent, and adding a key would be wasted work.
- Reads change nothing, so they're already idempotent — retry a GET as many times as you like. Under HTTP's rules (RFC 7231), GET, HEAD, PUT, and DELETE are defined as idempotent; POST is not, which is exactly why POST is the method that needs an idempotency key.
- Set-to-a-value writes are idempotent by shape: 'set the shipping address to X' has the same result whether it runs once or three times, because it overwrites rather than accumulates. It's the increment-style operations — 'charge $50', 'add one item', 'append a row' — that duplicate on retry and need a key.
- Operations that carry a natural unique id can dedup on that instead of a separate key: an order already has an order id, so 'create order #1234' can use #1234 as its own idempotency key. You only mint a fresh key when the request has no natural unique identity of its own.
The honest framing: reach for an idempotency key when an operation has a side effect that duplicates on retry and has no natural unique id to dedup on. That's most write-money and write-once operations, and almost no reads.
Three ways to get effectively-once#
The idempotency-key pattern is one of a few ways to make duplicates harmless. They differ in who holds the dedup state and how much the caller has to do. Here's the menu:
| Approach | How it dedups | Best for |
|---|---|---|
| Natural idempotency (PUT / set-to-value) | The operation overwrites, so a repeat is a no-op — no store needed | Writes that set a value rather than accumulate |
| Idempotency key + dedup store | Client tags the request; server saves key → result and replays it | Side-effecting POSTs: payments, orders, sending a message |
| Idempotent producer / transactions (Kafka) | The system stamps each message and dedups for you behind the API | Streaming pipelines that want effectively-once without hand-rolling keys |
They're not rivals so much as the same idea at different layers. Natural idempotency is free when your operation happens to have that shape. An idempotency key is what you add when it doesn't. And a system like Kafka's idempotent producer is that same key-and-dedup pattern built into the infrastructure, so application code doesn't have to carry it. When in doubt: if the operation overwrites, you're already done; if it accumulates and has a side effect, you need a key or a system that supplies one.
In the wild
- Stripe exposes idempotency directly: you send an Idempotency-Key header (a random string you generate, up to 255 characters — they recommend a UUID) on any POST, and Stripe stores the key and its response for 24 hours, replaying the saved result for repeats. Reusing a key with different request parameters is rejected, which stops a key from being accidentally reused for a different charge.
- Kafka's idempotent producer (enable.idempotence=true) gives each producer a producer id and stamps every message with a per-partition sequence number that counts up. The broker remembers the last sequence number it saw and silently drops any message whose number it has already accepted — so a producer retry after a lost acknowledgement doesn't write the record twice. That's an idempotency key and a dedup store, built into the broker.
- AWS SDKs retry failed calls automatically with exponential backoff and jitter built in, and many AWS APIs accept a client request token — their name for an idempotency key — so an automatic retry of, say, launching an instance doesn't launch two.
- Payment processors and banking APIs almost universally require an idempotency key on money-moving endpoints, for the reason this whole page is about: on a payment, a duplicate isn't a cosmetic bug, it's someone charged twice.
Pitfalls & gotchas
The client retried but generated a new idempotency key each time. What happens?
The dedup breaks completely. The whole pattern depends on the retry carrying the same key as the original, so the server can match them. A fresh key per attempt looks like a fresh operation every time, so every retry charges the card again. The key must be generated once, before the first send, and reused for every retry of that same operation — generating it inside the retry loop is the classic bug.
What if the server crashes after doing the work but before saving the key?
Then the retry finds no key, redoes the work, and you've double-charged — the exact failure the pattern exists to prevent. The fix is to make the work and the key-save atomic: write the charge and the dedup record in one database transaction, so either both commit or neither does. If saving the key can't share a transaction with the side effect, you record the key first as 'in-flight', do the work, then mark it 'done' — and a recovery step reconciles anything left in-flight.
Two identical requests arrive at the same instant with the same key. Doesn't the dedup store handle that?
Not on its own — that's a race. If both requests read the store before either writes, both see 'key not found' and both do the work. A plain read-then-write dedup is only safe for retries that arrive one after another. To be safe against simultaneous retries you need an atomic first-writer-wins check — typically a unique constraint on the key column, so the database admits exactly one insert and the loser is told to wait and replay.
Isn't 'exactly-once delivery' a feature some message queues advertise?
They advertise exactly-once processing, which is a different claim. Delivery over an unreliable network can't be exactly-once — that's the Two Generals impossibility. What those systems do is at-least-once delivery plus dedup at the receiver, giving a result that is effectively-once. The marketing says 'exactly-once'; the mechanism underneath is always retry plus dedup.
QuizA team adds retries to their payment client to survive network blips, but customers start reporting occasional double charges during outages. Retries reuse the same idempotency key and the server has a dedup store. What's the most likely cause?
- Exactly-once delivery isn't enabled on their network.
- The dedup window is shorter than how long their client keeps retrying, so late retries find the key already deleted.
- They should stop retrying — retries always cause double charges.
- UUID keys occasionally collide, merging two charges into one.
Show answer
The dedup window is shorter than how long their client keeps retrying, so late retries find the key already deleted. — During an outage the client backs off and keeps retrying for a while — sometimes longer than the server's dedup window. When a retry finally lands after the window has expired, its key has already been deleted, so the server treats the retry as a brand-new charge and bills the customer again. The fix is to make the window longer than the client's maximum retry horizon. 'Exactly-once delivery' isn't a real network setting (that's the impossibility). Retrying isn't the problem — an idempotent server makes retries safe, when the window is sized right. And UUID collisions are astronomically unlikely; that's not it.
In an interview
Idempotency comes up any time the design moves money, sends something, or writes across a network — payments, notifications, order placement, message queues. The interviewer is checking whether you know that retries are unavoidable and that duplicates are therefore a design problem, not an edge case. Lead with that.
- State the ambiguity first: a timeout can't tell you whether the request succeeded, so the client must retry, so the server will see duplicates. Everything follows from that one fact.
- Name the pattern crisply: client-generated idempotency key, reused on every retry; server-side dedup store mapping key → saved response; replay the response on a repeat. Say that saving the response (not just a flag) is what lets the retry get the same answer.
- Nail the 'exactly-once' trap: exactly-once delivery is impossible (Two Generals); you build effectively-once as at-least-once delivery plus an idempotent receiver. Interviewers ask this specifically to see if you'll claim the impossible.
- Bring up the two costs unprompted: the dedup window must outlast the client's retry horizon, and simultaneous retries need an atomic first-writer-wins check (a unique constraint), not a plain read-then-write.
- Mention the retry side: exponential backoff plus jitter to avoid a retry storm, a retry cap, and a circuit breaker — safety (idempotency) and rate (backoff) are two separate problems and a good answer covers both.
- Ground it in a real system: Stripe's Idempotency-Key header with a 24-hour window, or Kafka's idempotent producer using a producer id plus a sequence number.
PredictAn interviewer asks: 'You're designing a notification service. An upstream event can be delivered to you more than once, and each event should send exactly one email. How do you guarantee that?' What's the strong answer?
Hint: You can't stop the duplicates arriving. Where do you make them harmless, and what stops two copies racing?
Say up front that you can't get exactly-once delivery from the upstream, so you'll build effectively-once at your end: accept at-least-once delivery and dedup. Use a stable idempotency key from the event itself — an event id the producer already assigns — rather than minting a new one, so retries of the same event share a key. Keep a dedup store of keys you've already processed; on each event, atomically claim the key (a unique-constraint insert) before sending, so two copies racing in at once can't both send. Send the email only if you won the claim; if the key's already there, skip. Size the dedup window to outlast how long upstream might redeliver. Then add the retry side for your own outbound sends: backoff plus jitter so a mail-provider blip doesn't turn into a storm. The strong answer names both halves — dedup for safety, backoff for rate — and never claims exactly-once delivery.
References
- Stripe API — Idempotent requests — The Idempotency-Key header, the 24-hour window, and the saved-response replay, from the canonical implementation.
- RFC 7231 §4.2 — Safe and Idempotent Methods — The HTTP definition of which methods are idempotent (GET, PUT, DELETE) and why POST is not.
- Exponential Backoff And Jitter (AWS Architecture Blog) — Marc Brooker's write-up showing jitter, not just backoff, is what breaks up a retry storm.
- Kafka — Idempotent producer (KIP-98 / docs) — Producer id + per-partition sequence numbers: an idempotency key and dedup store built into the broker.
- The Two Generals' Problem — Why exactly-once delivery over an unreliable link is impossible — the theory under the timeout ambiguity.