Idempotency keys are a design input, not a retry hack

Most people meet idempotency as a retry hack: the network blipped, the client resent, and now there are two charges. So you bolt on a dedupe table and move on. That works until the day it matters — and then you discover the property you needed was supposed to be designed in, not sprinkled on.

The definition is boring and exact: an operation is idempotent if applying it twice has the same effect as applying it once. On a money path that isn’t a nicety. It’s the property that lets a queue retry, a consumer crash mid-batch, or an on-call engineer redrive a thousand parked messages at 2am — all without double-charging anyone.

Where the key comes from

The whole thing hinges on one question: what makes two requests “the same”? Get the key wrong and you either dedupe things that were meant to be distinct, or fail to dedupe the retries you were trying to collapse.

There are two honest answers, and they’re not interchangeable:

• Natural keys — the request already contains something unique and meaningful: a provider’s event ID, a payment intent ID, a (lease_id, period) pair for a monthly charge. Prefer these. They survive client retries, server restarts, and replays, because they’re derived from the domain, not from this attempt.
• Synthetic keys — the client generates a UUID and promises to reuse it on retry. Fine for client-initiated writes, useless for anything the client can’t see (a webhook, a scheduled job). A synthetic key is a contract with the caller; a natural key is a fact about the world.

The rule I follow: if the event has a natural key, use it. Reach for a synthetic one only when the caller is the sole source of identity.

The check and the effect commit together

This is the part people skip, and it’s the only part that actually matters. “Check if we’ve seen this key, then do the work” is two statements, and the gap between them is a race. Two retries arrive at once, both read “not seen,” both proceed, and you’ve charged twice — now with a dedupe table that confidently says everything is fine.

The check and the effect have to be atomic. Insert the key and apply the side effect in the same transaction. If the insert conflicts, the work never runs. If the work fails, the key rolls back with it.

// apply.ts
await db.tx(async (t) => {
  const seen = await t.insertIdempotencyKey(cmd.key); // UNIQUE constraint does the work
  if (!seen.inserted) return; // already applied — no-op, safe to replay

  await t.appendEvents(aggregate, cmd.events);
  await t.enqueueOutbox(cmd.events);
});

The uniqueness constraint on the key column is the real lock. The if (!seen.inserted) return is just how you read its answer. No advisory locks, no “check-then-act” — let the database arbitrate, because it’s the one thing that can.

Scope, and how long you keep it

A key is only meaningful within a scope. A payment-provider event ID is globally unique; a (merchant, invoice) pair is unique per merchant. Encode the scope into the key or the constraint, or you’ll get phantom collisions across tenants — the worst kind, because they look like successful dedupes.

Retention is the other quiet decision. Keys can’t live forever; the table would grow without bound. But delete them too early and a late retry sails through as “new.” Tie retention to the window in which a retry is plausible — usually the provider’s redelivery window plus a safety margin — and make it explicit, not a cron nobody remembers.

If processing an event twice can’t equal processing it once, you don’t have a queue. You have a liability with good latency.

What it is not

Idempotency is not deduplication after the fact, and it’s not “exactly-once delivery” — that’s a distributed-systems fairy tale. It’s effectively-once processing built on at-least-once delivery: the network may deliver your message one or five times, and your write side makes all but the first a no-op.

Designed in, it disappears into the architecture. You stop thinking about retries as a hazard and start treating them as free, because they are. The DLQ can redrive a batch and the keys make the replay boring. That’s the goal — failure handling that’s allowed to be aggressive precisely because it can’t double-apply.

Filed under systems. Got a key-derivation horror story? Tell me.