Idempotency Patterns: Building Retry-Safe Distributed Systems

Your payment API times out after 30 seconds. The client retries. Did the first request succeed? If it did and you process the retry, the customer gets charged twice. Timeouts don't tell you whether the request failed before or after the server processed it.

sequenceDiagram
    participant C as Client
    participant API as API server
    participant DB as Database<br/>(unique key constraint)
    C->>API: POST /payments<br/>Idempotency-Key: K
    API->>DB: BEGIN TX
    API->>DB: INSERT idempotency_keys(K) — claim
    alt First request (key new)
        DB-->>API: OK
        API->>DB: charge customer
        API->>DB: UPDATE idempotency_keys SET response = ...
        DB-->>API: COMMIT
        API-->>C: 201 Created
    else Retry (key exists)
        DB-->>API: UNIQUE_VIOLATION
        API->>DB: SELECT response WHERE key=K
        DB-->>API: stored 201 + body
        API-->>C: replay stored response (same status, same body)
    end

The diagram is the safety property in one picture: the unique-constraint on the idempotency key + same-transaction insert is the only correct way to make this race-free. A SETNX-then-process-then-commit flow has a TOCTOU window between the two steps that lets concurrent retries both succeed in claiming the key.

Why Exactly-Once Delivery Is Impossible

The Two Generals Problem^{[Akkoyunlu et al., 1975]} proves that no protocol can guarantee agreement over an unreliable channel — any acknowledgment can itself be lost. In practice, systems choose between:

What vendors call "exactly-once" is at-least-once delivery plus idempotent processing. The broker guarantees each message is delivered ≥1 times. Your application guarantees processing the same message twice produces the same result. Together: the effect is exactly-once.

Default to at-least-once and make every handler idempotent.

The Idempotency Key Pattern

The full request lifecycle — what the server does with the Idempotency-Key header on every retry:

graph TD
    Req[Request arrives<br/>Idempotency-Key: uuid-1234] --> Lookup{INSERT key<br/>with status=processing<br/>ON CONFLICT DO NOTHING}
    Lookup -->|inserted| New[New request<br/>process body]
    Lookup -->|conflict — already exists| Existing{Status of<br/>existing row?}
    Existing -->|completed| Return[Return stored<br/>response_code + response_body]
    Existing -->|processing| Wait[Return 409 Conflict<br/>caller should poll]
    Existing -->|failed| Replay[Return stored error<br/>do NOT re-process]
    New --> Apply[Apply business effect<br/>charge card, send email, etc]
    Apply -->|success| Save[UPDATE row<br/>status=completed<br/>store response]
    Apply -->|failure| SaveFail[UPDATE row<br/>status=failed<br/>store error]
    Save --> Resp[Return response<br/>+ same response on every retry]
    SaveFail --> Resp
    Return --> Resp
    Replay --> Resp
    style Return fill:#dfd
    style Save fill:#dfd
    style Replay fill:#fdd
    style Wait fill:#ffd

Two production rules visible in the flow: (1) INSERT ... ON CONFLICT DO NOTHING is the atomic "first writer wins" primitive — no separate SELECT-then-INSERT race; (2) processing status returns 409 (not the cached response) so the caller polls until the original request completes.

Client generates a UUID for each operation and sends it in a header. The server stores the key with the response and returns stored results on retries^{[Stripe idempotency]}.

Schema:

CREATE TABLE idempotency_keys (
    key VARCHAR(64) PRIMARY KEY,
    status VARCHAR(20) NOT NULL DEFAULT 'processing',
    request_path VARCHAR(255) NOT NULL,
    response_code INT,
    response_body JSONB,
    created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
    expires_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + INTERVAL '24 hours'
);

TTL tradeoff: At 10K req/s, each additional day of storage costs ~415 GB (512 bytes/row). 24-hour TTL is safe for most clients; go longer only if you have batch clients that retry once daily.

The Race Condition: Why Caching Isn't Enough

Checking Redis for the key, then processing the payment, then storing the key leaves a gap. Two concurrent retries can both read "key not found," and both proceed to charge the customer.

The only correct solution: insert the idempotency key and execute the business operation in the same database transaction.

Go: Idempotent Payment Handler

func (h *PaymentHandler) processPayment(ctx context.Context, idempotencyKey string, req PaymentRequest) error {
    tx, err := h.db.Begin(ctx)
    if err != nil {
        return fmt.Errorf("begin tx: %w", err)
    }
    defer tx.Rollback(ctx)
 
    // Claim the idempotency key — unique constraint prevents duplicates
    _, err = tx.Exec(ctx,
        `INSERT INTO idempotency_keys (key, status) VALUES ($1, 'processing')`,
        idempotencyKey,
    )
    if err != nil {
        return fmt.Errorf("duplicate request: %w", err) // unique constraint violation
    }
 
    // Execute the business operation
    payment := &Payment{ID: uuid.New(), Amount: req.Amount, Status: "completed"}
    _, err = tx.Exec(ctx,
        `INSERT INTO payments (id, amount, status) VALUES ($1, $2, $3)`,
        payment.ID, payment.Amount, payment.Status,
    )
    if err != nil {
        return fmt.Errorf("insert payment: %w", err)
    }
 
    // Store result for future replays
    body, _ := json.Marshal(payment)
    _, err = tx.Exec(ctx,
        `UPDATE idempotency_keys SET status = 'completed', response_code = 201, response_body = $1 WHERE key = $2`,
        body, idempotencyKey,
    )
    if err != nil {
        return fmt.Errorf("store response: %w", err)
    }
 
    return tx.Commit(ctx)
}

Three operations happen atomically: claiming the key, creating the payment, storing the response. If any step fails, the entire transaction rolls back. The database's unique constraint on idempotency_keys.key is the safety mechanism.

The Transactional Outbox Pattern

Publishing an event to Kafka after the database commit is a dual-write problem: commit succeeds, Kafka publish fails, downstream systems never learn about the payment.

The outbox pattern writes the event to a database table in the same transaction as the business operation. A background poller reads the outbox and publishes to Kafka.

@Transactional
public Payment processPayment(PaymentRequest req, String idempotencyKey) {
    // Claim idempotency key and create payment in same transaction
    outbox.claimIdempotencyKey(idempotencyKey);
    Payment payment = payments.save(new Payment(req.amount()));
 
    // Write event to outbox — same transaction
    outbox.save(OutboxEvent.builder()
        .aggregateId(payment.getId())
        .eventType("payment.completed")
        .payload(new PaymentCompletedEvent(payment.getId(), req.orderId()))
        .build());
 
    return payment;
}

A background poller publishes pending events:

@Scheduled(fixedDelay = 500)
public void publishPending() {
    for (OutboxEvent event : outbox.findPending(100)) {
        kafka.send(event.getTopic(), event.getAggregateId(), event.getPayload());
        event.markPublished();
    }
}

If Kafka publish fails, the event stays in the outbox and retries. If Kafka succeeds but the status update fails, the event publishes again — which is fine, because the consumer is idempotent.

Consumer-Side Deduplication

Every consumer must handle duplicates. Use the natural business key (payment ID), not the Kafka offset.

func HandlePaymentCompleted(ctx context.Context, event PaymentCompleted) error {
    tx, _ := db.Begin(ctx)
    defer tx.Rollback(ctx)
 
    // Dedup: use the natural business key
    result, err := tx.Exec(ctx,
        `INSERT INTO processed_events (payment_id, processed_at)
         VALUES ($1, NOW()) ON CONFLICT (payment_id) DO NOTHING`,
        event.PaymentID,
    )
    if err != nil {
        return err
    }
 
    // If the insert was a no-op (duplicate), skip processing
    if result.RowsAffected() == 0 {
        return tx.Commit(ctx) // already processed
    }
 
    // Update order status — same transaction
    _, err = tx.Exec(ctx,
        `UPDATE orders SET status = 'paid' WHERE id = $1 AND status = 'pending'`,
        event.OrderID,
    )
    if err != nil {
        return err
    }
 
    return tx.Commit(ctx)
}

Kafka offsets are not stable across rebalances. Payment IDs are globally unique and permanent. ON CONFLICT DO NOTHING makes the insert idempotent; RowsAffected() == 0 signals a duplicate.

Naturally Idempotent Operations

Some operations are inherently safe to retry:

Operations that set absolute state are naturally idempotent. Operations that modify relative to current state (increment, append) are not. Instead of balance = balance + 100, store the transaction with a unique ID and compute the balance from the log.

Production Checklist

• Idempotency key required: Make the header mandatory on endpoints that modify state (payments, orders).
• Check before rate limiting: Query the idempotency store before applying rate limits. Replays are free.
• TTL-based cleanup: Delete expired keys periodically. A 24-hour TTL is safe for most clients.
• Database index on expiration: CREATE INDEX ON idempotency_keys (expires_at) speeds cleanup.
• Status state machine: Track PENDING, COMPLETED, FAILED to handle circuit breaker scenarios.
• Reaper for stuck keys: Delete PENDING keys older than your request timeout (e.g., 60s) to prevent permanent blocks after crashes.

The schema and Stripe-style idempotency middleware

The schema below is what every production idempotency store needs — content hash to detect "same key, different body" attacks, status state machine for in-flight de-dup, response payload cached so retries return the original answer byte-for-byte:

-- migration: 0042_idempotency_keys.sql
CREATE TABLE idempotency_keys (
    -- Composite primary key: an idempotency key is only unique within a tenant.
    -- Using a global PK invites cross-tenant collisions that look like security bugs.
    tenant_id        BIGINT  NOT NULL,
    idempotency_key  TEXT    NOT NULL,
 
    -- Content hash of the request body — protects against key-replay with a
    -- different payload. Stripe guards the same way: a key reused with
    -- different parameters returns an idempotency_error.
    request_hash     BYTEA   NOT NULL,
 
    -- State machine: PENDING → COMPLETED | FAILED. Reaper deletes stale PENDING.
    status           TEXT    NOT NULL CHECK (status IN ('PENDING','COMPLETED','FAILED')),
 
    -- Cached response so a retry replays exactly the original answer.
    -- COMPLETED rows MUST have these populated; PENDING rows leave them NULL.
    response_status  INTEGER,
    response_body    JSONB,
 
    created_at       TIMESTAMPTZ NOT NULL DEFAULT now(),
    completed_at     TIMESTAMPTZ,
    expires_at       TIMESTAMPTZ NOT NULL,
 
    PRIMARY KEY (tenant_id, idempotency_key)
);
 
-- The reaper queries this index every minute. Without it, the cleanup pass
-- becomes a sequential scan as the table grows past ~10M rows.
CREATE INDEX idempotency_expires_idx ON idempotency_keys (expires_at)
    WHERE status IN ('PENDING','COMPLETED');
 
-- Unique-by-hash check inside the same key — atomic upsert relies on this.
CREATE INDEX idempotency_hash_idx
    ON idempotency_keys (tenant_id, idempotency_key, request_hash);

The Go middleware that ties it together — single-statement upsert that's race-free across concurrent retries, returns the cached response on hit, runs the handler exactly once on miss:

package idempotency
 
import (
	"bytes"
	"context"
	"crypto/sha256"
	"database/sql"
	"encoding/json"
	"errors"
	"net/http"
	"time"
)
 
const headerKey = "Idempotency-Key"
 
func Middleware(db *sql.DB, ttl time.Duration) func(http.Handler) http.Handler {
	return func(next http.Handler) http.Handler {
		return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
			key := r.Header.Get(headerKey)
			if key == "" || r.Method == http.MethodGet {
				next.ServeHTTP(w, r)
				return
			}
 
			tenantID := tenantFromContext(r.Context())
			body, _ := readAndReplaceBody(r)
			hash := sha256.Sum256(body)
 
			// Atomic insert-if-absent. Returns COMPLETED row's response on
			// duplicate; otherwise inserts PENDING and lets the handler run.
			var status string
			var respCode sql.NullInt32
			var respBody []byte
			err := db.QueryRowContext(r.Context(), `
				INSERT INTO idempotency_keys
				    (tenant_id, idempotency_key, request_hash, status, expires_at)
				VALUES
				    ($1, $2, $3, 'PENDING', now() + $4::interval)
				ON CONFLICT (tenant_id, idempotency_key) DO UPDATE
				SET tenant_id = EXCLUDED.tenant_id  -- no-op; forces RETURNING
				RETURNING status, response_status, response_body, request_hash = $3
			`, tenantID, key, hash[:], ttl.String()).Scan(&status, &respCode, &respBody)
			if err != nil {
				http.Error(w, "idempotency check failed", http.StatusInternalServerError)
				return
			}
 
			switch status {
			case "COMPLETED":
				w.Header().Set("Idempotency-Replay", "true")
				w.WriteHeader(int(respCode.Int32))
				_, _ = w.Write(respBody)
				return
			case "PENDING":
				// In-flight retry — refuse fast. The original request is still
				// running and will land its response when it completes.
				http.Error(w, "request in progress", http.StatusConflict)
				return
			}
 
			// FAILED rows: re-run the handler. The next ON CONFLICT will
			// transition us back to PENDING, which is what we want.
			rec := &captureRecorder{ResponseWriter: w, body: &bytes.Buffer{}}
			next.ServeHTTP(rec, r)
			finalize(r.Context(), db, tenantID, key, rec.status, rec.body.Bytes())
		})
	}
}
 
func finalize(ctx context.Context, db *sql.DB, tid int64, key string, status int, body []byte) {
	final := "COMPLETED"
	if status >= 500 {
		final = "FAILED" // do not cache 5xx — retry might succeed next time
	}
	_, _ = db.ExecContext(ctx, `
		UPDATE idempotency_keys
		   SET status = $1,
		       response_status = $2,
		       response_body = $3::jsonb,
		       completed_at = now()
		 WHERE tenant_id = $4 AND idempotency_key = $5
	`, final, status, json.RawMessage(body), tid, key)
}
 
// captureRecorder, readAndReplaceBody, tenantFromContext omitted for brevity —
// straightforward http.ResponseWriter wrapping.
var _ = errors.Is // avoid unused import warning in this excerpt

The two details that matter: (1) the response is cached only on success or 4xx (5xx is treated as transient — a retry might legitimately succeed), and (2) the in-flight PENDING returns 409 Conflict instead of waiting, because letting clients block on the in-flight request makes a slow primary turn into client-side retry storms.

The reaper that prunes stuck and expired rows — runs every minute, deletes in batches so a long-held PENDING doesn't escalate into a permanent block:

-- reaper.sql — schedule with pg_cron or an external job scheduler.
-- Two passes: stuck PENDING (crashed handlers) and expired COMPLETED.
WITH stuck AS (
  SELECT tenant_id, idempotency_key
    FROM idempotency_keys
   WHERE status = 'PENDING'
     AND created_at < now() - INTERVAL '60 seconds'
   LIMIT 1000
), expired AS (
  SELECT tenant_id, idempotency_key
    FROM idempotency_keys
   WHERE expires_at < now()
   LIMIT 1000
)
DELETE FROM idempotency_keys k
 USING (SELECT * FROM stuck UNION ALL SELECT * FROM expired) targets
 WHERE k.tenant_id = targets.tenant_id
   AND k.idempotency_key = targets.idempotency_key;

Schedule it with pg_cron (paste-ready), so the reaper runs whether or not your application's job scheduler is healthy:

SELECT cron.schedule(
  'idempotency-reaper',
  '* * * * *',
  $$ DELETE FROM idempotency_keys
      WHERE (status = 'PENDING' AND created_at < now() - INTERVAL '60 seconds')
         OR expires_at < now() $$
);

• Correlate retries: Add idempotency key as a span attribute in tracing middleware so all attempts of the same logical operation are queryable.
• Outbox poller every 500ms: Fast enough for most services. CDC (Debezium) only needed for >10K events/s.

Idempotency at the Database Layer

The middleware caches responses, but the actual idempotent write still has to happen somewhere — and the database is the only layer with strong enough primitives to make it correct under concurrent retries. Three patterns dominate, and the difference between them is how they handle the read-modify-write race.

The naive SELECT-then-INSERT is wrong even with READ COMMITTED isolation: two concurrent workers each see "row not present," each issues INSERT, and only one fails on the unique constraint — the other already executed side effects. SERIALIZABLE fixes correctness but converts the hot path into a serialization-failure retry loop that can starve under load. The two patterns that actually work in production:

-- Pattern 1: UPSERT with WHERE guard. Atomic in a single statement —
-- the WHERE on the UPDATE branch makes the operation a no-op if the
-- row was already finalized by an earlier retry.
INSERT INTO orders (order_id, customer_id, amount, status, created_at)
VALUES ($1, $2, $3, 'placed', now())
ON CONFLICT (order_id) DO UPDATE
   SET status = EXCLUDED.status
 WHERE orders.status = 'pending'   -- <-- merge-with-where: do not overwrite final state
RETURNING xmax = 0 AS inserted;     -- xmax=0 means we won the race

The xmax = 0 trick is Postgres-specific but invaluable: it lets the application distinguish "I inserted this row" from "I conflicted with an existing row" without a second roundtrip. MySQL's equivalent is checking ROW_COUNT() after INSERT ... ON DUPLICATE KEY UPDATE.

For operations where the unique business key is computed (e.g. "transfer X from A to B at time T"), use a Postgres advisory lock to serialize the critical section without blocking unrelated rows:

func (s *TransferService) Execute(ctx context.Context, t Transfer) error {
    tx, err := s.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted})
    if err != nil {
        return err
    }
    defer tx.Rollback()
 
    // Hash the natural key into a 64-bit advisory lock id.
    // Transaction-scoped: released automatically on COMMIT or ROLLBACK.
    lockID := xxhash.Sum64String(fmt.Sprintf("transfer:%s", t.NaturalKey()))
    if _, err := tx.ExecContext(ctx, `SELECT pg_advisory_xact_lock($1)`, int64(lockID)); err != nil {
        return fmt.Errorf("acquire advisory lock: %w", err)
    }
 
    // Now safe to read-modify-write — we hold the lock for this natural key
    // until the transaction ends. Concurrent retries with the same key block here.
    var exists bool
    if err := tx.QueryRowContext(ctx,
        `SELECT EXISTS(SELECT 1 FROM transfers WHERE natural_key = $1)`,
        t.NaturalKey()).Scan(&exists); err != nil {
        return err
    }
    if exists {
        return tx.Commit() // already done — replay returns success
    }
 
    if _, err := tx.ExecContext(ctx,
        `INSERT INTO transfers (natural_key, source, dest, amount) VALUES ($1, $2, $3, $4)`,
        t.NaturalKey(), t.Source, t.Dest, t.Amount,
    ); err != nil {
        return err
    }
    return tx.Commit()
}

Advisory locks scale better than row-level locks for this case because the row you want to lock does not exist yet — there is nothing to SELECT FOR UPDATE. The lock id is a hash of the natural key, so two unrelated transfers never block each other; two retries of the same transfer always do.

Edge Cases the Spec Doesn't Cover

Production failures cluster in three places the original RFC draft on idempotency keys glosses over:

Partial failure mid-handler. The handler claims the key, charges Stripe, then the pod is OOM-killed before writing the response. The next retry sees status PENDING for 60 seconds (the reaper threshold), gets 409 Conflict, and the client backs off. Once the reaper deletes the row the retry creates a new key — and charges Stripe a second time. The fix is not making the reaper smarter; it is recording the external side effect inside the same database transaction as the key claim. Stripe's API supports this directly via their own idempotency key (you forward yours into theirs), so on retry Stripe replays its cached response and the second "charge" becomes a no-op even if your local state was lost.

Idempotency key collisions. Two clients generate the same UUID — astronomically unlikely with v4 UUIDs but inevitable when a buggy SDK ships hard-coded keys, or a load test reuses a fixture. Without the request_hash column from the schema above, the second caller silently receives the first caller's response. With it, the server can detect "same key, different body" and return 422 Unprocessable Entity with {"error":"request_signature_mismatch"}. The hash should be salted per tenant so an attacker cannot probe whether a particular key is in use.

Key rotation across deploys. A common failure during deploy: v1 of the handler stores responses in one schema, v2 in another, and a retry that lands during the rollout reads a response_body shape its parser does not understand. The defence is a schema_version column on idempotency_keys plus a deserializer that knows how to read every version it has ever shipped. When the parser sees an unknown version it returns a fresh response from a re-execution rather than serving corrupt cached data — this is safe only because the underlying business operation is itself idempotent thanks to the database-layer pattern above. Without that property, the retry would double-charge.

The unifying lesson: idempotency is a layered guarantee. The HTTP middleware deduplicates network retries, the database layer deduplicates concurrent in-process retries, and the external API's own idempotency key deduplicates after-the-commit failures. Drop any one of the three and the system has a window where a retry causes a duplicate side effect.

Frequently Asked Questions

Keep Reading

• Building Resilient Distributed Systems with Go — Circuit breakers, bulkheads, and retry policies that complement idempotency
• Event-Driven Microservices in Go — The outbox pattern and idempotent consumers in Kafka
• Database Indexing Strategies — Indexing idempotency key expirations and dedup tables for production scale