Distributed Idempotency for Payments: The Three Pits and the Three-Layer Defense
Payment double-charges and confusing failure states produce customer complaints and support load. This layered design gives Western developers a concrete blueprint for making a single logical action happen exactly once across distributed systems.
Distributed payment idempotency breaks down in three predictable places. The first is check-then-write in application memory, where two concurrent requests both read the order as unpaid and both deduct. The second is separating token consumption from business success, leaving a window for retries to double-charge. The third is relying on Redis as the final judge when its async replication can lose the token in a failover.
The defense stacks three layers. The entry layer uses Redis SET NX with the order number as the key and a Lua script to atomically consume the token. The business layer uses a state machine UPDATE with WHERE status=0; an affected row count of 0 means someone else already handled it, so the duplicate request returns the correct outcome instead of failing. The data layer adds a unique index on the order number in the payment flow table, so a second insert hits a duplicate key and rolls back the transaction.
A more advanced pattern promotes the idempotency record itself to a table with status (processing, success, failure). This enables result replay for duplicate requests and handles the edge case where the gateway succeeded but the local write failed. The outbox pattern keeps notification messages in the same local transaction as the payment state change, so downstream failures don't stall the money path.
The atomicity requirement between checking and marking is the real crux; every layer is just a different way to close that gap.
Redis's async replication makes it fundamentally unsuitable for final idempotency decisions; it can only be a fast pre-filter.
Defining idempotency as 'duplicate requests get correct results' rather than 'reject duplicates' aligns the error handling with user-facing correctness.
The advanced idempotent-record table treats idempotency as a stateful entity, enabling result replay and integration with outbox, which a simple token cannot do.