WeChat's 10pm step-count push: 100 million notifications, one minute, and a lot of sharding
A push that looks like one simple read becomes a 100-million-row scheduling and write problem when it goes out to everyone at once. The fixes, sharding, queue buffering, hash-staggered delivery, and on-demand ranking, transfer directly to any large-scale notification or personalized feed.
Every night after 10pm, WeChat sends a step-count notification to hundreds of millions of users. Delivery times drift by about a minute on purpose: if every message fired at exactly 22:00:00, 100 million tasks would land in the same second. Even spread across 60 seconds, the average send rate is 1.67 million pushes per second, and that is only the delivery side.
A single machine scanning a remind_at table cannot carry that load. At 5,000 sends per second, 100 million notifications take about 5.5 hours, with no retry and no horizontal scaling. The workable shape is to separate scheduling from sending: shard tasks by uid, hash users into 60 send slots, enqueue uid batches, and let consumers pull at their own rate with rate limiting on the consumer side.
Personalized friend leaderboards make the compute side worse. A global ZSet handles one shared ranking, but one ZSet per user at an average 200 friends means 20 billion entries and a 200x write amplification on every step update. Sorting on query is cheap for small friend lists, but k has a long tail up to about 10,000, so the deciding factor is whether k times access frequency justifies caching. The last lever is to skip work entirely: if the notification only shows your own steps, no ranking is computed until someone opens the message, cutting precomputed leaderboards from 100 million to roughly 10 million at a 10% open rate.
The one-minute delivery window is the product decision that makes the system tractable. If users were not willing to accept a notification 'a little after ten', there would be no room to hash users into 60 send slots and the whole send path would have to absorb a single 100M-message spike.
Designing for average friend counts is a trap. The long tail of k, up to about 10,000 friends, means an account with 8,000 friends costs 40x more to read than an average one, and those are often the accounts whose leaderboards matter most.
The cost of precomputed friend leaderboards is dominated by write amplification, not storage, because step data arrives all day. On-demand sorting wins by moving that cost into a single read that often never happens.
Skipping work beats optimizing work. Cutting precomputation from 100M to 10M rankings saves the same order of magnitude as making 100M computations 10x faster, and it also avoids the machines and money spent on the unused 90%.
Delivery-time observations dominate the reply section: the 10pm push is ragged by a minute or two, a 22:21 delivery occurs, and step counting reportedly stops at 21:50. That raggedness fits the sharded, staggered scheduler the article describes, but a pointed question asks whether the visible delay comes from scheduling spread or client-side display latency. The article's own strategy of sharded schedulers, queue buffering, and on-demand leaderboard generation is restated in detail, while a contrarian line argues for client-side ranking and another joke answer proposes one server per user. The single-node versus cluster premise is left unresolved, and the caveat that the two environments produce opposite conclusions undercuts any universal architecture claim.
This article breaks down the implementation difficulty of the WeChat Sports 10pm leaderboard push from a technical angle: 1. For billion-scale pushes, sharded peak-staggering plus scheduler enqueueing and consumer queue buffering solves the single-machine scheduled-task performance bottleneck; 2. Instead of the high-cost pre-stored ZSet approach, the friend leaderboard weighs real-time computation or short caching based on friend count and access frequency; 3. There is no need to compute everything in advance; combined with the open rate, compute on demand, shifting pressure from the sending side to the reading side and adding corresponding optimizations. The core is reducing unnecessary computation.
The 10pm push isn't uniform—it can be off by one or two minutes. That looks like users are sharded and scattered, rather than a single top-of-the-hour job hitting everyone at once. If it really pushed to everyone simultaneously, the leaderboard read at that instant would be a sharp spike. Later in the article, do you attribute this variation to sharding, or is the client's receiving time itself skewed? I want to verify whether it's scheduling spread or display delay.
I want to confirm a premise: does this push-leaderboard system run on a single machine or a cluster? Our conclusions for the two environments are completely opposite.