跪拜 Guibai
← Back to the summary

A 180B-Parameter Model on 32GB: Expert Streaming Makes It Work, But Watch the Memory Peak

Suppose you have an internal log that isn't convenient to upload, and you want the local model to find the first failing call. The IDE and browser are still open on the laptop; the model takes a while to read the log, and you just want it to give you one verifiable clue.

Tarek Ziadé's experiment provides a reference for exactly this scenario. He ran Qwen3.8-Flash-Next, with 180 billion parameters, on an M5 MacBook with 32 GB of unified memory. After optimization, the generation speed for short answers approached 14 tokens per second. An earlier approach processing a document of about 6,000 tokens had device memory peak at 25.4 GiB, forcing the browser to be closed to avoid macOS starting to use disk as swap. Tarek Ziadé's experiment log

What I care about more is what happens when the model goes back to daily work: how much disk still needs to be free, how much memory remains, and how long it takes to get a useful result.

Select experts first, then read weights

Qwen3.8-Flash-Next is an MoE (Mixture of Experts) model. Each MoE layer has 512 selectable routing experts. When processing one token (a piece of text), the router selects 10 of them to participate in computation. Each layer also has shared experts that always participate.

A direct implementation might load layer by layer, moving all the expert weights for a layer at once. For this model, each layer would require moving 512 routing expert weights, of which only 10 actually participate. Tarek's approach is to let the router select experts first, then fetch those few weights. If there's a cache, use it; if any is missing, supplement it from the SSD. While computing the current layer, it can also try to prefetch experts likely to be needed in the next layer. Transformers experiment implementation

The router selects the experts needed for the current layer, checks the memory cache first, and reads missing weights from SSD on demand

For this Mac, CPU and GPU share unified memory, so moving weights to the CPU side still occupies the 32 GB. Putting experts that are not currently needed on the SSD frees up space.

Experts on SSD, but memory still goes to long documents

Tarek's early approach quantized the experts to about 3 bits (using fewer bits to store weights), taking about 49 GB on disk, with a device-side expert cache set to 12 GiB. The model also has an n-gram embedding table of about 51 GB (storing vectors for short sequences), also left on the SSD and read row by row on demand. Together with other weights that stay resident in memory, the actual footprint after loading is about 18 GiB. Weight configuration for the early approach

These numbers answer three different questions. Whether the file fits on disk, how much memory is left after loading, and where the peak goes when processing real input. The 25.4 GiB mentioned earlier comes from a test with about 6,000 tokens, which includes temporary space needed for handling long documents. Looking only at memory usage after startup would underestimate that headroom optimistically.

Short-output speed, converted to task wait time

Tarek continued to switch to smaller expert weights and adjusted GPU scheduling and prefetch. On the same 32 GB M5, Transformers' generation speed went from 10 tokens per second to 13.9. After reading a document of about 6,000 tokens, generation speed went from 7.3 to 12.4 tokens per second. In another test with about 4,000 tokens of input, the optimized prefill (processing input text) speed reached 228 tokens per second. Same-machine comparison data

At 13.9 tokens per second, generating a 200-token diagnosis alone takes about 14 seconds. I can accept that ten-plus seconds of output time for just inspecting a log. A programming agent usually reads files, changes code, runs tests, and then decides the next step based on new output. If it goes through five rounds, each generating 200 tokens, generation alone takes over a minute, with input reading and tool execution time on top. Tarek eventually left the major programming work on a desktop. Tarek Ziadé's usage judgment

Use one log to decide whether to keep this model

Pick a log with a known fault cause, open the browser and IDE the way you normally do, and run a small model that fits entirely in memory and this approach separately on the same question. Run only one at a time, record the full time and memory peak, and check whether the error locations they point to match.

If both models locate the fault accurately, I'd choose the one that gives a result faster. If the small model misses a key clue and the large model can point to a specific call and evidence, then spending a little more disk and wait time is worthwhile. Next time a similar problem shows up, I'll know which model to open.