Payment Callbacks Broke for Three Days Over a Milliseconds vs Seconds Default

Three days, five wrong directions, and the final culprit was a mismatch between default values for "milliseconds" and "seconds".

Figure 1 | The failure point was not in the payment service but in the gateway's timestamp window; it was the last thing we spotted in three days.

By the end you get three things: ① a 5-step breakdown method for tracing signature verification failures (no more guessing); ② why signature invalid is junk logging and how to turn it into logs that can locate the problem; ③ one thing to do before canary release: capture real samples instead of using requests you constructed yourself.

Five minutes total. Next time a callback returns 401, you can go through the checklist directly.

Background

• Architecture: gateway (Spring Cloud Gateway) + payment service + callback party
• Callback path: third party → gateway (signature verification, timestamp window, nonce anti-replay) → payment service
• Change: moved signature verification from the payment service to the gateway, making it a unified entry point

Symptoms

• After 5% canary, callback 401s jumped from 0 to over 30%, with near-total failure during peak
• The error was uniformly signature invalid, and the log had only one line: sign invalid
• Locally constructing requests with the same keys and parameters passed verification

First round: eliminating one by one

1. Keys: compared across the third-party admin console, gateway config, and payment service; consistent, eliminated
2. Concatenation rules: sorting per docs, null handling, URL encoding, byte-by-byte compared with captured raw request; consistent
3. Server clock: NTP sync normal, within 200ms deviation; eliminated
4. nonce: Redis write/read back normal; eliminated
5. Signature algorithm: SHA256WithRSA; cross-verified with a known correct sample; passed

Everything was correct, but production still failed.

Figure 2 | All five directions were eliminated; the real signal was 'all eliminated', not a reason to keep digging at the same point.

Stuck: could not reproduce locally

Locally constructed requests passed; copying the raw production capture into a local script also passed. This showed the problem was not in the "signed content" but in the environment where verification happens.

So two things were done:

• The verification-failure branch was split into separate instrumentation points: timestamp / sign / nonce judged separately
• A 1% canary was run and failure samples were saved (sanitized, keeping only field structure)

Root cause

100% of failure samples were stuck on the timestamp window:

• The gateway's timestamp unit configuration defaulted to seconds
• The callback SDK sent 13-digit millisecond timestamps
• The window check treated milliseconds as seconds, the difference instantly became 45000+, and everything was judged "expired"

The three whys made sense:

• Why local passed: local scripts used 10-digit second-level timestamps, inconsistent with production
• Why intermittent: during peak, another callback channel was used, with a different timestamp unit
• Why logs were useless: before instrumentation, only verification failed was logged, so it was impossible to see which segment got stuck

Figure 3 | Same callback, 10-digit and 13-digit differ by 1000x; local passed because local used 10-digit.

Fix

1. Changed the timestamp unit to explicit configuration, with automatic normalization for 10-digit / 13-digit
2. Emit WARN when unit mismatches, no longer failing silently
3. Signature verification failure logs are output by step, with sanitized samples
4. nonce write failures changed to observable degradation, no longer silently allowing or rejecting
5. Canary switch plus a signature verification success rate metric, with auto-rollback below 99%

Retrospective: three takeaways you can apply elsewhere

1. Every number in a protocol must explicitly declare its unit. Timestamps, timeouts, monetary precision, distances: if the docs don't state it, code has two interpretations, and one of the two interpretations is wrong.
2. signature invalid is junk logging. Signature verification is a composite process; failure must be split into steps, otherwise the troubleshooting cost is ten times higher.
3. Passing locally doesn't mean correct. Locally constructed requests use the format you assume, not the real production format. Capture real samples before canary.

Figure 4 | Three practices that can be carried directly into the next payment/gateway project.

Finally

This kind of problem has a common characteristic: it's not that one point is written wrong, but that the "default values" of two modules are inconsistent.

So now, when I do payment, gateway, or callback changes, the first step is not writing code; it's putting together a 'field and unit convention table' and getting confirmation from all parties before starting.

If you run into a similar problem, talk about it in the comments; if it can be reproduced, I'll reply.