跪拜 Guibai
← All articles
Architecture · Backend · AI

A 180B-Parameter Model on 32GB: Expert Streaming Makes It Work, But Watch the Memory Peak

By AAA爱学习的刘哥 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

A consumer laptop can now run a 180B-parameter model, but the trade-off is a memory peak that forces closing other apps and generation speeds that rule out multi-step agent workflows. For developers who need local log analysis without sending data out, this shows the practical limits of SSD-backed MoE inference.

Summary

An expert-streaming technique runs Qwen3.8-Flash-Next, a 180B-parameter MoE model, on a 32GB unified-memory M5 MacBook. Each MoE layer has 512 routing experts, with only 10 active per token. Instead of moving all 512 weights, the router picks the needed experts, checks a cache, reads missing weights from SSD, and prefetches the next layer's likely experts. The early configuration quantized experts to about 3 bits, leaving ~49GB on disk, a 12 GiB device-side expert cache, and a ~51GB n-gram embedding table on SSD read row by row. Resident memory after load sits around 18 GiB.

The memory peak is the real constraint. A ~6000-token document pushed device memory to 25.4 GiB, forcing the browser to close to keep macOS from swapping to disk. After swapping smaller expert weights and adjusting GPU scheduling and prefetch, generation speed rose from 10 to 13.9 tokens per second for short answers, and from 7.3 to 12.4 after reading the same ~6000-token document. Prefill speed on a ~4000-token input reached 228 tokens per second.

Those numbers translate into waiting time. At 13.9 tokens/s, a 200-token diagnosis takes ~14 seconds, acceptable for log inspection. A coding agent doing five rounds of 200-token generation spends over a minute on generation alone, plus input and tool execution. Tarek kept primary coding on a desktop, so this setup suits occasional, single-query analysis, not interactive agent workflows.

Takeaways
— Each MoE layer in Qwen3.8-Flash-Next has 512 routing experts; the router activates only 10 per token.
— Tarek Ziadé's implementation runs the router first, then loads only the needed expert weights, using a cache and SSD reads.
— Early configuration quantized experts to ~3 bits, totaling ~49GB on disk, with a 12 GiB device-side expert cache.
— A ~51GB n-gram embedding table stays on SSD and is fetched row by row.
— Resident memory after load is about 18 GiB, but a ~6000-token document pushed peak memory to 25.4 GiB.
— Generation speed after tuning reached 13.9 tokens/s on short answers and 12.4 tokens/s after reading a 6000-token document; prefill hit 228 tokens/s on a ~4000-token input.
— Generating a 200-token response at 13.9 tokens/s takes about 14 seconds.
— Five rounds of 200-token generation for a coding agent would exceed a minute, not counting input and tool execution.
Conclusions

The memory peak from long inputs, 25.4 GiB, is the actual usability limit, because swapping to disk on unified memory degrades everything else running on the machine.

A token rate of 13.9 per second looks respectable until multiplied across multiple agent steps; a five-step agent burns over a minute just generating text, making the approach unsuitable for interactive coding.

Tarek's own choice to leave coding on a desktop indicates the trade-off is acceptable only for narrow, single-query tasks like log analysis, not for workloads with many dependencies.

Concepts & terms
MoE (Mixture of Experts)
A model architecture where each layer has many specialized sub-networks (experts) and a router selects a small subset per token, reducing compute and memory compared to a dense model of the same total parameter count.
Expert streaming
A loading strategy that fetches only the expert weights the router selects, caching them and reading missing ones from SSD on demand, instead of loading all experts for a layer into memory.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗