Payment Callbacks Broke for Three Days Over a Milliseconds vs Seconds Default
A unit mismatch between two services can make every valid callback fail, and the symptom looks like a signature or crypto bug, not a data format bug. Teams building payment or webhook integrations should treat field-unit agreements as part of the contract before canary traffic; otherwise `sign invalid` logs send them down every wrong path first.
Payment callback signature verification moved from the payment service into a Spring Cloud Gateway as a unified entry point. At 5% canary traffic, callback 401s went from 0 to over 30%, approaching total failure at peak. The logs carried one line: `sign invalid`, and locally reconstructed requests passed. Keys, concatenation rules, server clock, nonce, and the SHA256WithRSA algorithm all checked out.
Local reproduction stalled the hunt. Even raw production captures replayed locally passed, which pointed to the verification environment rather than the signed payload. The failure branch was then instrumented into timestamp, sign, and nonce segments, and a 1% canary saved sanitized failure samples. Every failure sat in the timestamp window: the gateway defaulted to second timestamps while the callback SDK sent 13-digit milliseconds, so the window check saw a difference over 45000 and rejected everything as expired.
Fixes included explicit timestamp unit configuration with automatic 10/13-digit normalization, WARN logs on unit mismatch, stepwise verification logs, observable nonce-write degradation, and auto-rollback when verification success fell below 99%. The incident rules: every protocol number needs an explicit unit, composite verification needs stepwise failure logs, and local passes prove nothing unless test data matches production's actual format.
The five clean checks did more than clear suspects. They showed the signed payload was fine in every local sense, and the failure lived in the verification environment; that distinction made the next step possible.
A single log line saying `sign invalid` costs hours because signature verification is a composite process. Splitting the log into timestamp, sign, and nonce segments turned a three-day hunt into a direct lookup.
Default values are a hidden contract. A gateway and an SDK can both handle timestamps correctly under their own assumptions and still be 1000x apart; the fix is explicit unit agreement, not more debugging.
Local reproduction can confirm your own model of the request and still miss production reality. Captured traffic from a 1% canary exposed the mismatch that hand-built test data could not.