AIGC · AI Programming

The $4.1M Reality Check: What It Actually Costs to Run a 2.8-Trillion-Parameter MoE Model On-Prem

By 李剑一 · Jul 27, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

The gap between API subscription pricing and the capital cost of on-prem AI hardware is wide enough to reshape build-vs-buy decisions for entire engineering teams. A 30-million-yuan upfront investment breaks even against 30 monthly API subscriptions only after a century, which makes the cloud the rational default for all but the largest enterprises or most sensitive workloads.

Summary

A full on-premises deployment of a 2.8-trillion-parameter Mixture of Experts model like Kimi K3 is an enterprise-scale engineering project, not a single-server job. The minimum viable configuration starts with 64 H200 GPUs, each with 141GB of VRAM, spread across eight server chassis with dual-socket Xeon or EPYC CPUs and 1-2TB of DDR5 memory per node. The cluster also demands NVLink switches, 800G optical modules, and RDMA-capable Ethernet to handle the high-frequency data synchronization between experts that MoE architectures require during inference.

Cooling 64 high-wattage cards packed into a single cabinet rules out air cooling; cold-plate or immersion liquid cooling is mandatory, along with enterprise-grade titanium power supplies to handle a single-cabinet draw that can reach 160kW. Storage needs are equally heavy, with RAID arrays of at least 8TB per server to hold the 1.4TB model weights and KV cache.

A conservative hardware estimate lands at 30 million yuan (about $4.1 million) as a one-time capital expense. Operating costs add roughly 910,000 yuan ($125K) per year in electricity alone, assuming 24/7 full load at industrial power rates and a PUE of 1.55. The calculation excludes consumables, network transit, and data-center rental, making it a floor, not a ceiling.

Takeaways

— A minimum viable local deployment of a 2.8T-parameter MoE model requires 64 H200 GPUs with 141GB VRAM each, spread across eight server chassis.

— NVLink switches and 800G optical modules are necessary to prevent the expert-to-expert data synchronization from bottlenecking inference.

— Each server needs dual-socket Xeon Sapphire Rapids or AMD EPYC Genoa CPUs, plus 1-2TB of DDR5 memory to hold the KV cache and intermediate activations.

— Cold-plate or immersion liquid cooling is mandatory because 64 H200 cards in one cabinet overwhelm any air-cooling setup.

— Conservative one-time hardware cost is 30 million yuan ($4.1M), excluding server-room renovation, consumables, or network transit.

— Annual electricity cost alone runs about 910,000 yuan ($125K) at 24/7 full load, assuming industrial power rates and a PUE of 1.55.

Conclusions

The cost breakdown makes a blunt case that on-prem deployment of frontier MoE models is an infrastructure play, not a software decision. The networking and cooling requirements alone gate it behind data-center-grade facilities.

Comparing the 30-million-yuan hardware outlay against 30 monthly API subscriptions at 699 yuan each yields a break-even horizon of roughly 119 years, which collapses the build-vs-buy argument for nearly every team.

The power calculation exposes a hidden truth about AI TCO: electricity alone costs more per year than the annual salary of a junior engineer in many markets, even before accounting for the hardware depreciation.

Concepts & terms

Mixture of Experts (MoE)

A model architecture that activates only a subset of its total parameters (the 'experts') for any given input, reducing computation per token but requiring high-bandwidth communication between experts during inference.

NVLink

NVIDIA's high-bandwidth, low-latency interconnect technology that allows GPUs to communicate directly at speeds far beyond standard PCIe, critical for multi-GPU inference workloads like MoE models.

PUE (Power Usage Effectiveness)

A data-center efficiency metric calculated as total facility power divided by IT equipment power. A PUE of 1.55 means that for every watt used by the servers, an additional 0.55 watts is consumed by cooling, lighting, and other overhead.

KV Cache

A memory buffer that stores the key and value tensors from previous tokens during autoregressive generation, avoiding recomputation and speeding up inference at the cost of significant memory consumption, especially with long context windows.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗