跪拜 Guibai
← Back to the summary

RTX 5090 vs. 128GB Unified Memory: What Local AI Actually Needs

theme: hydrogen highlight: xcode

local-ai-cover.png

Many people may not know that the RTX 5090 you bought for tens of thousands of yuan might not be able to run the local AI models you want to run. I'm not joking.

Suppose you're running local AI models at home. I'll give you two hardware options.

Which one would you choose?

Most people would choose the discrete graphics card without a second thought, since its compute is far greater than the on-board chipset. There's also another reason: the discrete card is expensive!!! You can't go wrong buying the expensive one, right?

To make it more intuitive, let's compare their compute power first. Below is single-precision 32-bit floating point compute (FP32).

Just looking at these FP32 numbers, the discrete card is 7 times more powerful than the on-board chipset.

Is compute power alone enough?

Let me pause and list what we know about both:

What you might not have expected is that when running local AI models, it's not clear which one is more suitable.

There's a counterintuitive situation: when a model doesn't fit in VRAM, the mini PC with an on-board chipset might actually be more suitable.

The problem lies in that 32GB of VRAM. As soon as the model gets a bit larger, the weights may not fit.

For example, a 70B-parameter model, with 4 bits per parameter, needs 35GB of weights (about 32.6 GiB). The RTX 5090's VRAM can't hold all the weights. Of course, you can still run it by putting part of them in system memory, but the speed suffers.

In contrast, the AMD mini PC has 128GB of on-board memory, which handles these weights with ease.

Wait, memory? That's not VRAM!

Here's the magic: the on-board chipset's memory is shared between the GPU and CPU, so it's called "unified memory." Its advantage is that you can allocate as much memory as possible to a single processor, so it can handle AI models with large memory consumption.

Apple's M-series chips have always used this unified memory architecture, which is one reason some people buy a Mac mini to run local models.

Of course, other vendors have followed suit. AMD's Strix Halo, NVIDIA's DGX Spark, some Intel Core Ultra and Qualcomm Snapdragon X models also allow CPU and GPU to share memory.

At this point, you might ask: since the on-board chipset has these advantages, is it still necessary to buy a discrete graphics card?

Here another memory metric emerges: bandwidth.

Weakness of the on-board chipset is precisely memory bandwidth.

So-called "memory bandwidth" refers to the speed at which memory delivers data to the processor. Although the RTX 5090 doesn't have much VRAM (32GB is already high!), its memory bandwidth is huge, reaching 1792GB/s, while AMD's bandwidth is only 256GB/s.

Let's assume a model has 40GB of weights, and we generate one token at a time; each token generation requires reading the weights once. The AMD machine has 256GB/s bandwidth; at full bandwidth it reads a little over 6 times per second (256 divided by 40). Just looking at bandwidth, that's about 6 tokens per second. Who can tolerate such a snail's pace?

With an Apple M3 Ultra with 819GB/s bandwidth, reading the same 40GB of weights would take about 20 reads per second. The RTX 5090 has even higher bandwidth, but those 40GB of weights don't fit in its VRAM.

So memory capacity and memory bandwidth are both bottlenecks for AI models. This is also why high-bandwidth memory (HBM) prices have gone through the roof.

If you have to read all these weights every time, the mini PC will run slowly.

But things are never absolute!

To save compute, the "Mixture of Experts (MoE)" architecture emerged. When computing a token, it doesn't need to read all parameters; it only activates a portion of them.

Take Qwen3-30B-A3B for example. It has 30.5B parameters, and for each token generated it activates only about 3.3B. With 4 bits per parameter (half a byte) roughly, the full set of weights is about 15.25GB (all 30.5B parameters), and the portion used each time is about 1.65GB (activated 3.3B). Divide AMD's 256GB/s by that: at full bandwidth, it can read over 150 times per second.

So if you're using a unified-memory mini PC, choose MoE models.

Of course, mini PCs have another big disadvantage: their compute power!

We know that in real-world model usage, the model must first process the user's prompt before generating tokens. Since the mini PC's compute is slow, processing prompts is also slow.

For example, running a 70B model on this AMD chip measured about 95 input tokens per second. If you paste a 4,000-token document, it takes over 40 seconds just to finish reading it.

Think about it: once users stuff more content into the context window, simply processing the prompt becomes minutes to tens of minutes.

In short, at this stage, running large models on a local computer has drawbacks whether you use a discrete GPU or an on-board chipset. At best, you can run some medium-sized MoE models, and the prompts can't be very long.

Why DeepSeek V4.1 Flash is so popular

Let's take another example. Can the 128GB AMD mini PC run DeepSeek V4.1 Flash?

Yes, it can.

I'm referring to version 1017, with about 763B total parameters (this is truly huge!). The full weights certainly won't fit into 128GB of memory.

But like the Qwen model earlier, it only activates about 16B parameters per token generated (and can be even smaller).

Using the same 4-bit calculation, 16B × 4 ÷ 8 = 8 GB. That's the weight size used to generate one token.

What about speed?

AMD's memory bandwidth is 256 GB/s. Assuming the 8 GB of weights used each time is in memory and bandwidth is fully utilized, 256 ÷ 8 = 32, meaning it can read 32 times per second.

What about the RTX 5090? Its VRAM bandwidth is 1792 GB/s. Similarly, 1792 ÷ 8 = 224, reading 224 times per second. That's pretty good!

That's why people who play with local models like models like DeepSeek V4.1 Flash.