How a Dual-Vendor MaaS Setup Cut P99 Latency 85% Under a 400x Load Spike

From 50,000 to 20 Million Daily Calls: Performance Engineering Practices for Large Model Inference Services

Preface

At the beginning of 2025, I took over an internal platform providing external large model API call services. At that time, it had just been online for three months, with a daily call volume of less than 50,000. The overall technical architecture was lightweight and simplified, only meeting the initial small-scale traffic verification needs: a single GPU server running a Qwen-7B model, wrapped with a layer of FastAPI. The P99 latency was stable at 4800ms, which felt acceptable.

Half a year later, this platform needed to handle an average of 20 million daily large model API calls, with peak QPS exceeding 1800. Under nearly 400 times the traffic pressure, the system's P99 latency actually dropped from 4800ms to 720ms.

The significant performance optimization was not accidental; it was the engineering result of multiple rounds of stress testing, failure reviews, and architectural iteration and reconstruction. The most critical decision among them was the dual-vendor architecture.

Phase 1: The Collapse of the Self-Built Solution and the Forced Pivot

The initial choice of self-built inference had core advantages in autonomy and controllability, no external API call fees, and fully customizable model parameters and inference logic. Initially, only a single GPU server was deployed, relying on the native transformers framework to load the Qwen-7B model, with each request independently executing generate inference.

When the daily call volume climbed from 50,000 to 500,000, systemic issues erupted: under a stress test scenario of 500 concurrency, P99 latency soared from 4.8 seconds to over 7 seconds, and GPU computing power was nearly at full load long-term. During one instance where a client submitted 200 inference requests in a batch, the lack of KV cache reuse and batch scheduling caused the model instance to run out of VRAM and crash directly.

We initially tried self-built optimization methods like model quantization, request rate limiting, and inference parameter tuning, but the performance improvement was limited. If we expanded the self-built GPU cluster, the cycle for hardware procurement, data center deployment, and approval processes would take months, a pace the explosive business traffic growth could not wait for. At this point, the architecture had only two viable paths: continue the heavy-asset expansion of the self-built cluster, or switch to mature MaaS inference services. We ultimately chose the latter, and decided to introduce two vendors simultaneously rather than relying on a single one.

Phase 2: Dual-Vendor Architecture — Balancing Cost, Performance, and Risk

In our architecture, the second vendor is not a cold standby but regularly shares traffic, undertaking the dual roles of load sharing + failover:

• Load Sharing: Daily, 20%-30% of batch tasks with high latency tolerance and non-core business traffic are routed to the second vendor. After diversion, the load on the primary vendor decreases, and the overall P99 latency is optimized simultaneously.
• Failover Disaster Recovery: When the primary vendor experiences a regional failure or severe performance jitter, traffic can be fully switched within minutes. The vast majority of users are unaware, with only a small portion of requests being processed needing to retry.
• Cost Leverage: Splitting traffic between two service providers creates a competitive dynamic. We can dynamically adjust traffic weights based on real-time pricing, resulting in a more advantageous overall TCO compared to a single vendor for the same service specifications.
• Fault Failover Guarantee: We regularly conduct failover drills during both off-peak and peak hours, actively shutting down the primary vendor's API access point to verify that the backup system can complete a full traffic cutover within 7 minutes.

Essentially, that 20%-30% of traffic already incurs inference costs. The difference is only whether it's all settled with a single vendor or split between two. Exchanging traffic splitting for bargaining power and high availability guarantees does not create idle redundant overhead out of thin air.

The cost is also real: we needed to self-develop a lightweight weighted scheduling layer (supporting health probing and dynamic weights), integrate two sets of APIs, maintain two monitoring dashboards, and conduct after-sales communication with two vendors separately. This brings ongoing operational overhead. But for a production system with 20 million daily calls, the ROI of this investment is far higher than the potential loss from a single point of failure.

Phase 3: Selection and Implementation of Dual Vendors

After clarifying the dual-vendor strategy, we spent three weeks investigating mainstream MaaS service providers and finally settled on two: Lanyun Yuanshengdai as the primary, and Alibaba Cloud Bailian as the failover and load-sharing layer.

Lanyun Yuanshengdai: The Primary Inference Node Supported by Measured Data

Lanyun carries 70%-80% of our core traffic, responsible for all latency-sensitive intelligent conversations and real-time inference. This choice was not made on a whim but based on long-cycle measured data from an authoritative third-party evaluation platform — AI Ping.

The core metric output by AI Ping evaluations is P90 latency, i.e., the response time achieved by 90% of all end-to-end requests. According to AI Ping's evaluation results from June 4 to June 11, 2026, Lanyun ranked among the top in throughput and P90 latency performance on the DeepSeek-V3.2 model; it was also in the industry's leading group for the Qwen series models. At the same time, it was one of the few among over 20 service providers that could stably control P90 latency within 1 second, with metrics significantly ahead of the second tier. Specific details are shown in the screenshots below for your reference. It is worth mentioning that recently both service providers have also adapted DeepSeek-V4, and we are preparing to switch our core business over.

Figure: Throughput data for Qwen3-235B-A22B on the AI Ping platform, 2026.6.4-6.11

Figure: Latency data for Qwen3-235B-A22B on the AI Ping platform, 2026.6.4-6.11

Figure: Throughput data for Deepseek-V3.2 on the AI Ping third-party evaluation platform, 2026.6.4-6.11

Figure: Latency data for Deepseek-V3.2 on the AI Ping third-party evaluation platform, 2026.6.4-6.11

From an engineering perspective, Lanyun provides a standard OpenAI-compatible API. We only changed two lines of configuration, base_url and api_key, to connect. This low migration cost allows us to flexibly schedule traffic between the two service providers.

Weakness: It adopts a curated model operation strategy and does not bulk-list a massive number of niche open-source models. However, our production business stably uses only the DeepSeek and Qwen series, for which it has instead performed deep performance tuning, perfectly matching our business needs. A large model library count does not equate to suitability for one's own business scenarios; the ultimate optimization capability for core scenarios is the key selection criterion.

Alibaba Cloud Bailian: The Stable Cornerstone for Failover and Load Sharing

Alibaba Cloud Bailian's positioning in our architecture is very clear: a stable, compatible, multi-region failover and load-sharing layer.

The reasons for choosing it are very pragmatic:

• Alibaba Cloud's data centers cover the entire country, allowing for nearby access and reducing network latency.
• Bailian has excellent compatibility with mainstream models like Qwen and DeepSeek, with almost zero adaptation cost.
• The team is most familiar with Alibaba Cloud's monitoring and operations system, enabling quick problem localization.

As a leading domestic cloud provider, its mature enterprise-level service guarantees and emergency response mechanisms provide reliable support for the overall architecture's failover stability.

Phase 4: The Final Architecture's Operating Mechanism and Results

After half a year of evolution, our final architecture's operational logic includes multiple layers of optimization:

1. High-Performance Inference Base: The MaaS vendors' underlying layer uses professional inference engines like vLLM/TensorRT-LLM, replacing the inefficient self-built transformers.
2. Dual-Vendor Traffic Splitting and Peak Shaving: The global peak QPS exceeded 1800. After traffic splitting, the load pressure on each of the two vendors was significantly reduced, avoiding performance collapse triggered by a single vendor being fully loaded.
3. Dynamic Routing and Health Probing: A self-developed lightweight scheduling service continuously probes the P90 latency (referencing AI Ping metrics and internal monitoring) and error rates of both vendors, automatically adjusting traffic weights; when a vendor's latency exceeds a threshold or a regional failure occurs, traffic is quickly switched.
4. Elastic Cluster Resource Pool: The vendors possess massive GPU clusters, eliminating single-server VRAM/computing power bottlenecks and supporting auto-scaling.

The final results brought by this set of mechanisms are illustrated by internally monitored P99 data:

Metric	Before Optimization	After Optimization
P99 Latency	4800ms	720ms
GPU Resources	Single GPU long-term full load	Elastic scaling, no self-built hardware
Max Daily Calls	Approx. 300,000 (bottleneck)	Over 20 million

Special Note: 4800ms was the P99 latency when the daily call volume was 50,000 (internal monitoring data), and 720ms is the measured P99 when the daily call volume is 20 million with a peak QPS of 1800+. Traffic increased 400-fold, yet latency decreased by 85% — this is the result of multiple layers of optimization including a professional inference base, dual-vendor traffic splitting, dynamic routing, and elastic clusters.

As for third-party evaluation dimensions, AI Ping's P90 data provided a key basis for our selection, while in the production environment, we focus more on P99 to guarantee the long-tail user experience.

Q&A: The Two Most Common Questions About the Dual-Vendor Architecture

Q1: A dual-vendor setup sounds good, but our company has a limited budget. Under what circumstances is it actually worth doing?

A: Based on our practical experience, a dual-vendor setup is a rational architectural investment when the following three conditions are met simultaneously:

1. You have enough total traffic to split: At least a million daily calls. Diverting 20%-30% to a second vendor still represents a "meaningful business" for them, allowing you to get a normal price; simultaneously, this 20%-30% itself is a valid payload (like batch tasks with high latency tolerance, non-core business), not idle.
2. You have the capability for traffic scheduling: It doesn't need to be complex; a weight-based routing layer plus health checks is sufficient.
3. You are willing to accept the operational cost of dual vendors: Integrating two sets of APIs, two monitoring systems, and two after-sales services in exchange for disaster recovery capability and bargaining power.

If the traffic is too small (e.g., daily calls in the tens of thousands) or all business is extremely latency-sensitive (cannot be split), then honestly choosing the best single vendor is more cost-effective.

Q2: Besides looking at third-party evaluations, what other practical selection methods do you have?

A: In addition to referencing long-cycle P90 data from platforms like AI Ping, we internally insist on doing three things: real traffic replay stress testing, production A/B testing, and fault failover drills.

Keep a close eye on the "performance degradation rate," and don't be fooled by low-concurrency peaks — focus on the degradation multiple of P99 latency and throughput at 50 and 100 concurrency relative to single concurrency. The closer the degradation rate is to 1, the more "stable" the system is under high pressure.

Furthermore, two additional special reminders:

• First, the AI Ping platform only provides P90 latency data and does not provide P99 latency. If you need to evaluate long-tail latency (P95/P99), you must build your own stress testing environment and simulate high-concurrency scenarios with real business traffic. Do not directly use AI Ping's P90 data as P99.
• Second, some MaaS service providers impose a request rate limit (Rate Limit) on each API Key by default. Taking Lanyun as an example, when we first connected, we found we could only call it about 20 times per second, far below business needs. This type of restriction is an industry default and is usually not prominently marked in official documentation. You need to proactively contact customer service to apply for a limit increase before stress testing, otherwise the stress test data will be completely distorted due to rate limiting.

Conclusion

From 50,000 to 20 million daily calls, from 4800ms to 720ms P99 latency — behind this seemingly counter-intuitive result lies a rational abandonment of the self-built path, real attempts at intermediate optimization paths, and, more importantly, a commitment to the architectural philosophy that "dual vendors are not a waste."

Although dual vendors also bring additional operational costs and the complexity of integrating two systems, it is the division of labor between dual vendors, being online simultaneously, dynamic allocation, supplemented by a self-developed scheduling layer and health probing, that has created this inference architecture capable of withstanding tens of millions of traffic while maintaining extremely fast responses.

If you are also making architectural choices for large model inference, it is recommended to first look at the continuous monitoring data on AI Ping (focusing on P90 latency), then be sure to conduct your own stress testing to obtain P99 metrics, and communicate with vendors about rate limits in advance. After all, under real production pressure, an architecture without a backup is essentially gambling.