跪拜 Guibai
← All articles
Artificial Intelligence

128GB Unified Memory Makes Local LLMs a Commodity, but the Real Fork in the Road Is Software

By 挖掘狂人 ·
Read original on juejin.cn ↗ Google Translate ↗ Alt translation

For developers sizing up local inference, the practical takeaway is blunt: if you want long-context 100B models on the desktop, 128GB unified memory is the only threshold, and the bandwidth race is already over. The next battleground is software that decides what runs locally, what goes to the cloud, and how agents route between them.

Summary

The wall for running large models locally is VRAM: a 24GB GPU cannot fit a 70B model once you add the KV cache for long context. The form factor that actually crosses that threshold is a mini PC with 128GB of unified memory, where CPU and GPU share one memory pool. Tests show that a 100B-class model can consume 80-100GB, which is impossible for a 24GB card but routine for a 128GB machine.

Memory capacity also dictates context length. Ollama defaults to 4k context under 24GB, and only opens 256k above 48GB. On the model side, Qwen3.6's 35B MoE runs at 20 tokens per second on a 64GB laptop's integrated GPU, though it tends to lose its way on long multi-step tasks. Measured alongside each other in November, an NVIDIA DGX Spark and an AMD-platform mini PC produced nearly identical generation speeds (273 vs 256 GB/s bandwidth, a "rounding error"); the difference is capacity, not velocity. The AMD machine costs about half as much and runs the full x86 software stack, while the Spark offers faster inference peaks and CUDA.

Since October, system makers worldwide have converged on this 128GB unified memory form factor, and Microsoft added AMD's Ryzen AI Max to its local model support list. Hardware is becoming an entry ticket. The real differentiation now sits in software: Codigger proposes three layers — local inference for high-frequency tasks, cloud workstations for peak compute, and agent orchestration that automatically routes tasks between the two, so users never shuffle files by hand.

Takeaways
— A 24GB VRAM GPU cannot run a 70B-parameter model locally because the KV cache for long context adds significant memory beyond the weights.
— 128GB of unified memory can allocate the vast majority of its pool to the GPU, making it the only practical desktop form factor for 100B-class models with long context.
— A 100B-class model can consume 80-100GB of memory for weights plus KV cache.
— Ollama's default context windows are tiered: 4k under 24GB, 256k only above 48GB, so memory directly controls how much conversation a local model can remember.
— Qwen3.6's 35B MoE version runs at 20 tokens per second on a 64GB laptop's integrated GPU, but the community reports it tends to get lost on long multi-step tasks.
— In November, an NVIDIA DGX Spark and an AMD-platform mini PC with 128GB unified memory showed nearly identical generation speed (273 vs 256 GB/s bandwidth, called a 'rounding error').
— The AMD mini PC costs about half the price of the DGX Spark and supports the full x86 software stack, while the DGX Spark offers faster inference peaks and CUDA.
— Since October, multiple system makers have entered the 128GB unified memory mini PC form factor, and Microsoft added AMD's Ryzen AI Max to its local model support list.
— Codigger's proposed software stack has three layers: local inference for high-frequency tasks, cloud workstations for peak compute, and agent orchestration that automatically routes tasks between local and cloud.
— The local-plus-cloud workflow with agent orchestration is designed to avoid users manually moving files between environments and to keep data in-domain for local tasks.
Conclusions

The 273 vs 256 GB/s bandwidth difference being called a 'rounding error' suggests that memory bandwidth in this class of unified memory devices has effectively plateaued for current model workloads.

The real constraint for local LLMs is no longer raw compute speed but memory capacity and the software that manages context and task routing.

Microsoft's formal support for AMD's Ryzen AI Max signals that the 128GB unified memory mini PC is becoming a mainstream target for local model deployment, not just a niche enthusiast toy.

Codigger's three-layer approach (local, cloud, agent) indicates that the winning software will treat the data-center and the desktop as two ends of the same pipeline, rather than separate silos.

The report that Qwen3.6's 35B MoE gets lost on long tasks suggests that memory capacity alone doesn't guarantee reliable multi-step reasoning; stability is a separate quality that depends on model architecture and context handling.

Concepts & terms
Unified memory
A memory architecture where the CPU and GPU share a single memory pool, allowing the GPU to allocate much more than a typical VRAM ceiling. In a 128GB unified memory machine, most of that pool can be assigned to the GPU.
KV cache
A cache that stores the key-value representations of tokens processed so far in a transformer model. Its size grows with context length and can exceed the model weights for long contexts, making it a major memory consumer.
MoE (Mixture of Experts)
A model architecture that activates only a subset of its parameters per token, allowing higher total parameter counts without proportionally increasing inference compute. The Qwen3.6 35B MoE version uses this to run efficiently on limited hardware.
Source: juejin.cn ↗ Google Translate ↗ Backup ↗