A 180B-Parameter Model on 32GB: Expert Streaming Makes It Work, But Watch the Memory Peak
A consumer laptop can now run a 180B-parameter model, but the trade-off is a memory peak that forces closing other apps and generation speeds that rule out multi-step agent workflows. For developers who need local log analysis without sending data out, this shows the practical limits of SSD-backed MoE inference.
An expert-streaming technique runs Qwen3.8-Flash-Next, a 180B-parameter MoE model, on a 32GB unified-memory M5 MacBook. Each MoE layer has 512 routing experts, with only 10 active per token. Instead of moving all 512 weights, the router picks the needed experts, checks a cache, reads missing weights from SSD, and prefetches the next layer's likely experts. The early configuration quantized experts to about 3 bits, leaving ~49GB on disk, a 12 GiB device-side expert cache, and a ~51GB n-gram embedding table on SSD read row by row. Resident memory after load sits around 18 GiB.
The memory peak is the real constraint. A ~6000-token document pushed device memory to 25.4 GiB, forcing the browser to close to keep macOS from swapping to disk. After swapping smaller expert weights and adjusting GPU scheduling and prefetch, generation speed rose from 10 to 13.9 tokens per second for short answers, and from 7.3 to 12.4 after reading the same ~6000-token document. Prefill speed on a ~4000-token input reached 228 tokens per second.
Those numbers translate into waiting time. At 13.9 tokens/s, a 200-token diagnosis takes ~14 seconds, acceptable for log inspection. A coding agent doing five rounds of 200-token generation spends over a minute on generation alone, plus input and tool execution. Tarek kept primary coding on a desktop, so this setup suits occasional, single-query analysis, not interactive agent workflows.
The memory peak from long inputs, 25.4 GiB, is the actual usability limit, because swapping to disk on unified memory degrades everything else running on the machine.
A token rate of 13.9 per second looks respectable until multiplied across multiple agent steps; a five-step agent burns over a minute just generating text, making the approach unsuitable for interactive coding.
Tarek's own choice to leave coding on a desktop indicates the trade-off is acceptable only for narrow, single-query tasks like log analysis, not for workloads with many dependencies.