Idempotency Keys: The Cheap Insurance Against Duplicate Writes

Shopify's webhook delivery model is explicit about the tradeoff: "at-least-once" delivery, not exactly-once and not guaranteed in order. Failed deliveries retry up to eight times over a four-hour window, and after that the webhook subscription is auto-removed until it's manually re-created. Deliveries can also arrive up to a day late. None of that is unusual — most webhook- and cron-driven systems make the same tradeoff, because "definitely delivered, maybe twice" is a lot easier to build than "exactly once."

What an unprotected retry costs

The problem is what happens on the other end. If a retry, or a slow duplicate delivery, reaches code that writes to a CRM, a payment processor, or an ad platform, and that write isn't protected, the result is a second contact record, a second charge, or a second order. The trigger did its job correctly. The handler just ran twice.

The fix is a key, not a queue

An idempotency key is a unique identifier for one logical operation — generated once, checked before the write happens, and reused on every retry of that same operation. It doesn't require rearchitecting how retries work; it just makes the second attempt provably safe.

Stripe's implementation is the clearest real-world reference for this pattern. A client generates a key, Stripe recommends a UUID v4, and sends it in the Idempotency-Key header on a POST request. Stripe stores the status code and body of the first request made with that key and returns the identical result on every later request with the same key, including replaying a stored 500 error, so a client retrying after a timeout can't accidentally create a duplicate charge. Keys expire after 24 hours.

Not every API being written to has this built in. When it doesn't, the equivalent gets built by hand: a small table keyed on your own idempotency key, checked before the write and updated after it, so a retried job sees "already done" instead of re-executing.

Where the key comes from matters

The key has to be generated at the point where an operation is decided on, not at the point where the API happens to get called — otherwise a retry just generates a fresh key and defeats the purpose. For webhook-triggered writes, Shopify already provides two headers built for exactly this: X-Shopify-Webhook-Id, unique per delivery attempt, and X-Shopify-Event-Id, shared across every retry of the same underlying merchant action. The second one is the one to key on, since it stays stable across retries by design. For cron-triggered writes, generate the key at the start of the job run, tied to whatever makes that operation logically unique — an order ID, a sync batch, a scheduled date — and persist it before making the call.

None of this is exotic. It's a few extra columns and a check-before-write. That's exactly why it's worth doing early. Once a system has been live long enough to produce real duplicate charges, emails, or orders, retrofitting idempotency means reconciling records that already exist, not just adding a check that should have been there from day one.