跪拜 Guibai
← Back to the summary

Distributed Idempotency for Payments: The Three Pits and the Three-Layer Defense

用户连点两次“确认支付” 怎么保证只扣一次?

Last week I had dinner with a friend who just came back from an interview at some company, looking exhausted.

I asked what happened. He said a question had crushed him.

The interviewer opened casually: "You said you've worked on payments. If a user double-clicks the confirm button on the payment page, how do you guarantee the same request is processed only once?"

He answered just as casually: "Add an anti-duplicate token. When the request comes in, check Redis first; if it's there, block it; if not, write it."

The interviewer nodded and asked: "What if two requests arrive simultaneously on two different machines?"

He froze: "Redis is shared, so it should block them, right?"

The interviewer pressed: "After the token check passes, the business is still deducting the amount. The second request comes in then. Have you deleted the token or not? If you delete too early and the business fails, what then? If you don't delete, what if the charge is duplicated?"

He fell silent.

This question appears very frequently in interviews for payment and transaction positions. It's the kind that "looks simple but collapses under follow-up." Too many people answer "store it with Redis," and interviewers have long grown tired of hearing that.

But honestly, those four words aren't wrong; they're just insufficient. Underneath lies three pitfalls, and those three pitfalls exactly cover all the difficulties of distributed idempotency.

Pitfall 1: Check-then-write is equivalent to no check under concurrency

The vast majority of people's first version of code looks like this:

Order order = orderMapper.selectByOrderNo(orderNo);
if (order.getStatus() == PAID) {
    return "已支付";
}
doPay(order);   // 扣款

Runs for a year without issues on a single machine with low concurrency. But if two requests collide even once out of twenty thousand, you get a double deduction.

The reason is simple: two threads execute selectByOrderNo simultaneously and both read "pending payment." Both think they're the first to arrive, both proceed, both deduct. The database never gets a chance to say "wait, this has already been handled"—because you never asked it to make that decision; the decision was made in application memory.

Some will say add a lock. On a single machine, synchronized works, but on two machines it doesn't. Use a distributed lock? That works, but lock granularity, renewal, timeout, and crash during the hold introduce a whole new set of problems, and it's still inherently "check-then-write"—just shrinking the window from milliseconds to microseconds. The window is still there.

What really needs to change isn't adding a lock; it's changing the approach:

"Determining whether something has been handled" and "marking that it has been handled" must be one indivisible action.

Once it's done in one step, there's no gap in between, and no one can slip through. That's called atomicity. Every solution that follows is essentially implementing this sentence at a different level.

Pitfall 2: You're blocking the request, not the deduction

The second pitfall is more hidden.

Many people implement it like this: when a request arrives, check the token; if it passes, delete the token, then start the business logic. Looks fine.

The problem is: at the moment you delete the token, the business hasn't succeeded yet.

What if the call to the third-party payment gateway times out right after? What if the local transaction rolls back? By then the token is already gone. The user retries, the request sails through, and you deduct again.

The reverse is also true. If you place the deletion after business success, then during the period between "token check passed" and "business success," both requests can pass the check, because you've split "consuming the token" and "executing the deduction" into two steps, leaving a door open in between.

This is what the interviewer really wants to hear when asking "did you delete the token or not": whether your idempotency boundary coincides with your business transaction boundary.

If they don't coincide, someone will always find a way to slip through the crack.

Pitfall 3: Redis is not absolutely reliable

Many candidates never even think of this layer.

Redis master-slave replication is asynchronous. The master returns success after SETNX, but the slave may not have synced yet. If the master crashes at that moment and the sentinel switches to the slave, the token you just wrote—gone.

So the retry request happily gets a new token and deduces again, with full justification.

The reverse direction has its own issue: if the "delete token" operation on the master hasn't synced and the slave is promoted, the token is still there, and duplicate requests get blocked incorrectly. That direction is safe (better to block than to deduct wrongly), but the user experience is "I clearly didn't pay successfully, but it says I submitted a duplicate."

The conclusion is direct: you can't stake the last line of defense for fund safety on an AP system.

Redis is suitable for "quick interception," not for "final judgment." The final say must be left to the database.


At this point, the logic for a three-layer solution naturally emerges. They're not three parallel tricks; they're three layers of depth defense: Redis blocks the first wave, the state machine blocks the second, and the unique index is the final backstop.

Layer 1: Entry layer, compress duplicate-check and reservation into one action

The standard approach is "token + SET NX".

When the user opens the payment page, the frontend first requests a token from the backend, and the backend writes it to Redis:

String key   = "pay:token:" + orderNo;
String token = UUID.randomUUID().toString().replace("-", "");

// SET key value NX PX 300000 —— NX means 'write only if not exists', this step itself is atomic
Boolean ok = redisTemplate.opsForValue()
        .setIfAbsent(key, token, Duration.ofMinutes(5));

return ok ? token : null;

Note two things. First, the key uses the order number, not a random string. This is a detail, but a critical one. If the idempotency key were a pure random token, when the user accidentally refreshes the page and the frontend requests a new token, both tokens would pass the check, and the anti-duplicate mechanism fails. The idempotency key must fall on the business dimension; one order has exactly one key.

Second, consuming the token must use Lua, not GET then DEL.

-- KEYS[1] = pay:token:{orderNo}
-- ARGV[1] = token passed by the client
local v = redis.call('GET', KEYS[1])
if v == ARGV[1] then
    redis.call('DEL', KEYS[1])
    return 1
end
return 0

GET and DEL are two separate commands; the window in between is enough for a second request to squeeze in, both read the token as existing, and both are allowed through. Lua scripts execute serially in a single thread in Redis, and the entire script runs without interruption—that's what's called atomic.

Why not just DEL and check the return value? Because DEL returns "how many keys were deleted," and it can't verify the token content. Anyone holding that order number could consume someone else's token. That's not anti-duplicate; that's handing over the keys.

A side trap worth mentioning: if you want to check something else in the Lua script (like a blacklist key), note that under Redis Cluster all keys must be in the same slot, otherwise you'll get a CROSSSLOT error. Use hash tags to pin the key prefix.

At this layer, only one of two concurrent requests gets the token; the other gets 0 and directly returns "Please don't submit repeatedly."

But this only blocks "same-time duplicates." It doesn't block retries, doesn't block Redis data loss, and doesn't block a user who waits ten minutes and clicks again.

Layer 2: Business layer, let the order status only move forward

After the token is granted, the payment logic truly begins. This layer doesn't rely on locks; it relies on a state machine.

Define states clearly, allow one-way flow, no going back:

WAIT_PAY(0) → PAYING(1) → PAID(2)
                ↘ FAILED(3)

The key is in the update: put the current state into the WHERE clause.

UPDATE pay_order
SET status = 1,
    version = version + 1,
    update_time = NOW()
WHERE order_no = 'P20260108001'
  AND status = 0
  AND version = 7;

Then look at the affected row count. affectedRows == 0 is the whole answer. It means the row is no longer in the state you thought it was—someone else already changed it before you looked. At this point, don't raise an error; query the latest status:

There's a point many get wrong, and I want to make it clear: affectedRows == 0 does not equal failure; it equals "this has already been done." The correct definition of idempotency is not "reject duplicate requests," but "let duplicate requests also get the correct result." The difference in user experience shows up in the number of complaints.

As for version, to be honest, in scenarios like payment where the status changes every time, the condition AND status = 0 already provides a current read with row lock, which is sufficient. Version's real value lies elsewhere: when your update doesn't change status but overwrites other fields (for example, when someone else is changing the amount or notes at the same time), without a version number you'd get silently overwritten.

Keeping both doesn't hurt, but don't think version is the key—WHERE status = 0 is.

Layer 3: Data layer, unique index as the final backstop

The last layer goes to the database.

Create a unique index on the order number in the payment flow table:

CREATE TABLE pay_flow (
    id          BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    order_no    VARCHAR(64) NOT NULL COMMENT '订单号',
    channel_no  VARCHAR(64) DEFAULT NULL COMMENT '渠道流水号',
    amount      DECIMAL(12,2) NOT NULL,
    create_time DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
    UNIQUE KEY uk_order_no (order_no)
) ENGINE=InnoDB;

Even if both previous layers are bypassed, when two requests reach the insert of the flow record, only one succeeds. The other hits a unique key conflict, the insert fails, the transaction rolls back, and the subsequent deduction code never runs.

This is the only place in the whole solution that "even if all application layers fail, it still won't fail." Because the constraint is enforced by the database itself, it doesn't depend on anyone writing correct code.

But there's a very real trap here, and many people step on it without realizing:

try {
    payFlowMapper.insert(flow);          // 撞唯一键
} catch (DuplicateKeyException e) {
    return "重复请求";                    // 然后在这里返回
}
// 同一个事务里继续干别的 → 抛 UnexpectedRollbackException

In a Spring transaction, once a DataIntegrityViolationException is thrown, the transaction is marked rollback-only. You catch the exception and the code continues, but the moment the transaction tries to commit, it throws an UnexpectedRollbackException, and the error message has nothing to do with the duplicate key, making debugging painful.

The correct approaches are two: either make the insert run in an independent transaction (REQUIRES_NEW), so failure is failure and has nothing to do with the outer transaction; or don't use exception flow control at all—use INSERT IGNORE or ON DUPLICATE KEY UPDATE and check the affected row count.

I personally lean toward the former; the semantics are clearer.


If the interviewer keeps digging, play this card

At this point, you've already answered. But if the other side wants something deeper, there's an idea worth discussing: manage the "idempotency record" itself as a stateful entity, not as a one-time cache switch.

The Redis token has a natural flaw: it's a binary thing—either it exists or it doesn't, and it carries no information about "how far the process has gone." So it can't answer a key question: did the request that arrived first actually succeed?

Switch to a table:

CREATE TABLE pay_idempotent (
    id          BIGINT UNSIGNED AUTO_INCREMENT PRIMARY KEY,
    biz_key     VARCHAR(64) NOT NULL COMMENT '业务幂等键,比如订单号',
    status      TINYINT NOT NULL DEFAULT 0 COMMENT '0处理中 1成功 2失败',
    result_code VARCHAR(32) DEFAULT NULL COMMENT '成功后回放给调用方的结果',
    expire_time DATETIME NOT NULL,
    UNIQUE KEY uk_biz_key (biz_key)
) ENGINE=InnoDB;

The flow becomes:

  1. When a request comes in, first INSERT a record with status = 0. Only the request that successfully inserts is qualified to call the payment gateway. If the insert fails, it means someone else is already processing.

  2. The request that failed to insert reads the record's status:

    • 0 Processing → return "Processing, please wait later"; don't return success, don't return failure
    • 1 Success → directly replay result_code to it; this is result idempotency
    • 2 Failed → allow retry
  3. After calling the gateway successfully and writing locally, update status to 1 in the same transaction.

This solves a problem the three previous layers didn't: what to do when the gateway succeeded but the local database write failed.

At this point the business is actually successful—the money is already out—but you just didn't record it. If you rely on "delete token to allow retry," the user's retry will deduct a second time. In the state machine approach, the retry sees 0 or 1, so it won't deduct again; it will just tell you "we're checking, wait."

As for records stuck at 0 and not moving, rely on expire_time and a reconciliation job to query the channel side actively, then fill in the result. You absolutely cannot simply delete the record and pretend it never happened—deleting it is equivalent to abandoning the memory of that money.

There's a companion idea worth mentioning as well: the Outbox pattern.

Payment results need to notify the order system, points system, risk control, message push, and so on. If one downstream fails, you shouldn't drag down the main money path. The approach is to write "what message to send" and "update payment status" in the same local transaction, store it in an outbox table, and let an asynchronous task deliver it. If delivery fails, it goes to a dead-letter queue for retry.

This way the main path only does one thing: get the deduction right. Notifications can be slower and heavier; that's fine.

Comments

Top 1 of 2 from juejin.cn, machine-translated. The original thread is authoritative.

用户580095225348

Awesome [grinning]

挖掘狂人

Thanks for the recognition, let's keep learning and improving together!