128GB Unified Memory Makes Local LLMs a Commodity, but the Real Fork in the Road Is Software
At the end of the year, anyone who wants to run large models at home or in the office hits the same wall: VRAM.
A graphics card with 24GB of VRAM cannot fit a 70B-parameter model — because besides the model itself, the long-context "memory cache" (KV cache) also consumes extra memory. To run models with tens of billions of parameters and long context locally, the form factor that actually has the headroom on desktop devices is the same kind: a mini PC with 128GB of unified memory.
Unified memory means the CPU and GPU share a single memory pool — most of the 128GB can be allocated to the GPU.
An analogy: a traditional computer is like a kitchen with a small separate storage room (VRAM); if the ingredients increase slightly, there is no room to move. Unified memory is like having one warehouse for the whole house, both sides can take from it freely. Running a 100B-class model, weights plus KV cache easily consume 80-100GB. This is a threshold that a 24GB graphics card can never reach, but it is an everyday occurrence for a 128GB machine.
How big is the threshold gap? Look at the measured tiers: Ollama (the most popular local model tool) has a default strategy — below 24GB it only gives 4k context, and above 48GB it dares to open 256k.
In other words, memory determines not only "can it run" but also "how much conversation it can remember". On the model side, there is also a downward push: the Qwen3.6 35B MoE version can run at 20 tokens per second on a laptop's integrated GPU with 64GB of memory. However, the community also reports that it tends to "get lost" on multi-step long tasks. The threshold is indeed lowering, but "enough capacity + stable speed" is still the baseline for serious tasks.
So what does the optimal solution look like? In November this year, two desktop devices with 128GB of unified memory sat on the same evaluation table: the NVIDIA DGX Spark and an AMD-platform mini PC.
The conclusion is interesting: their generation speeds were nearly identical (the 273 vs 256 GB/s bandwidth difference was called a "rounding error" by the reviewer), "what really makes them interesting is capacity, not speed". The differences lie elsewhere:
- NVIDIA DGX Spark: faster inference peak, CUDA ecosystem
- AMD mini PC: compatible with the full x86 software stack, price is about half, similar domestic machines in China are already in the 23,000 RMB range
Since October, domestic and foreign system makers have piled into this form factor, and Microsoft has officially added the AMD Ryzen AI Max to its "local model" support list — form factor convergence is already an industry consensus.
Form factor convergence precisely shows one thing: hardware is becoming an entry ticket, not a differentiator. 128GB devices will only become more numerous and cheaper. The real watershed is at the software layer: on this device, what kind of AI experience runs?
Codigger's answer consists of three things:
- Local inference handles high-frequency tasks: searching information, organizing documents, code assistance, done on local models, not burning tokens, data stays in-domain;
- Cloud workstations handle peaks: heavy compute for compilation, rendering, training goes to the cloud on demand, bills are predictable;
- Agent orchestration connects the two: tasks stay local when they should, go to the cloud when they should, and the Agent automatically schedules, instead of making users shuffle files between two environments.
This experience is not tied to a specific device: it is available in the cloud today and will also appear on all kinds of local compute devices in the future.
The pace of the hardware race is set by the whole industry; the answer for experience has to be written by software.
128GB of unified memory turns "running large models locally" from a geek toy into a daily necessity — and what determines whether it is good to use is always the software running on top of that memory.