DeepSeek-V4-Flash Can Run Locally, but You'll Need 128 GB of RAM
The release of DeepSeek-V4-Flash was truly explosive. Within 1 to 4 hours after its release, I noticed DeepSeek's response speed became significantly slower. I guess everyone was connecting to the API for testing. But it's no wonder—the official version of DeepSeek-V4-Flash even outperformed the DeepSeek-V4-Pro Preview. Its Agent capabilities have been greatly enhanced. It's incredibly impressive; what more could you ask for?
However, besides accessing the API, according to Unsloth AI, you can now run DeepSeek-V4-Flash locally.
DeepSeek-V4-Flash-0731, with 284B total parameters, 13B active parameters, and a 1M context window, now has a GGUF version available. You can run it on your own machine via Unsloth Studio or llama.cpp.
Image source: Unsloth AI
Sounds great, right?
But after reviewing the official configuration, I realized that "local" here is a bit different from what many people imagine... A minimum of 92GB total memory is required. To run a reasonably reliable version, it's recommended to start from 110GB...
Those with ordinary 32GB or 64GB computers shouldn't bother for now.
Let's first level the information playing field. According to the official announcement of DeepSeek-V4-Flash, its Agent capabilities have improved significantly compared to the Preview.
In 9 official Agent benchmark tests, it outperformed its predecessor Flash Preview and also beat the larger-parameter V4-Pro Preview. For example, Terminal Bench 2.1 jumped from 61.8 to 82.7, and DeepSWE rose from 7.3 to 54.4.
Image source: DeepSeek Official
DeepSeek-V4-Flash-0731 has many quantized versions, but you really only need to remember one thing: The lower the bit count, the smaller the footprint, but the more noticeable the quality loss.
The total memory here refers to system RAM plus VRAM, or unified memory on a Mac.
There's a catch: a file being only 103GB doesn't mean 103GB of memory is enough to run it.
After the model loads, KV Cache, context, and the system itself continue to consume memory. The official minimum is 110GB, but it's safer to just calculate based on 128GB.
There's another easy mistake to make.
Unsloth's original post labeled 4-bit as lossless. The current documentation now clarifies this: UD-Q4_K_XL is near-lossless, while the truly lossless version is UD-Q8_K_XL. The difference between the two is only 7GB.
Therefore:
- For a 128GB machine, choose
UD-IQ3_XXS. - For a 192GB or 256GB machine, go straight for
UD-Q8_K_XL. - If you only have 64GB, use the API first, or wait for smaller quantized versions.
Image source: Unsloth; The further to the bottom right, the closer the model is to the official original weights.
The Easy Way: Unsloth Studio
If you don't want to deal with a bunch of compilation parameters, you can just install Unsloth Studio.
For Mac, Linux, and WSL, open the terminal:
curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio -p 8888
For Windows, use PowerShell:
irm https://unsloth.ai/install.ps1 | iex
unsloth studio -p 8888
Then open in your browser:
http://127.0.0.1:8888
On the first launch, you will be prompted to create a password. Go to the Model hub, search for DeepSeek-V4-Flash-0731-GGUF, choose the quantized version based on your memory, and click Download.
Image source: Unsloth; The machine in the screenshot has 64GB VRAM plus 64GB RAM, and the 155GB Q4 version already exceeds its capacity.
The download size ranges from 100GB to 162GB, so it's recommended to reserve enough hard drive space in advance.
After the download is complete, just start it. Studio will automatically include the chat template and most inference parameters. You can also switch the thinking intensity in the bottom right corner: Non-think, Think High, and Think Max.
Image source: Unsloth
If you are only using it locally, do not add -H 0.0.0.0 to the startup command. This parameter allows other devices on the same network to access it. It's acceptable on a home intranet, but don't do this on the public internet.
If you want to connect your own tools, use llama.cpp
If you are already using llama.cpp, first make sure your version is up to date, then pull the model directly from Hugging Face.
For a 128GB machine, you can start with 3-bit:
./llama.cpp/llama-cli \
-hf unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-IQ3_XXS \
--temp 1.0 \
--top-p 1.0 \
--min-p 0.0 \
--ctx-size 32768
If you have enough memory and want to run the official lossless version, just change the model name:
unsloth/DeepSeek-V4-Flash-0731-GGUF:UD-Q8_K_XL
If the download keeps getting stuck, you can first install huggingface_hub and manually download the 3-bit file locally:
hf download unsloth/DeepSeek-V4-Flash-0731-GGUF \
--local-dir unsloth/DeepSeek-V4-Flash-0731-GGUF \
--include "*UD-IQ3_XXS*"
It is recommended to only open a 32K context for the first run. Once you confirm the model, speed, and memory are all normal, you can increase the context length.
Although it supports a 1M context, the longer the context, the greater the additional memory consumption.
Unsloth's general recommended parameters are as follows:
temperature = 1.0
top_p = 1.0
min_p = 0.0
context = start from 32768
If you are using it for Agent or coding tasks, change top_p to 0.95.
Think High is enabled by default. Use it for daily complex tasks first. Think Max requires at least a 384K context, and both memory usage and waiting time will increase further. It's suitable for when you really have a difficult problem. For ordinary Q&A, you can switch to Non-think for faster speed.
The command to turn off thinking mode is:
--chat-template-kwargs '{"enable_thinking":false}'
For Think Max:
--chat-template-kwargs '{"reasoning_effort":"max"}'
Image source: Unsloth; The speed in the official example is 49.7 tok/s. The specific speed depends on the hardware and quantized version.
So, running DeepSeek-V4-Flash locally is indeed possible. It's just that this "local" setup can only run on workstations and large-memory Macs for now...
For someone like me with a base-model 32GB Mac, I wonder if I'll ever be able to wait for it...
References: