← Back to the summary

Qwen 3.8-27B Lands: A Local 4-bit Model That Matches Opus 4.6 on Coding and Agent Benchmarks

Just last week, when DeepSeek-V4-pro, Grok 4.6, DeepSeek Harness, and GLM 5.3 were making waves one after another, Qwen 3.8 reached its open-weight moment.

While everyone was chasing after DeepSeek and GLM, Qwen also had its moment.

The reception for Qwen 3.8 has been unexpectedly good.

You can already see Qwen-3.8-2.4T and Qwen-3.8-27B on Hugging Face.

Then Unsloth provided a local deployment method for everyone.

This time it's not just theoretically possible to run locally — it really can run.

According to Unsloth's Qwen3.8 local deployment documentation, the 4-bit quantized version of Qwen 3.8-27B can start with 17–19GB total memory. RTX 4090, RTX 5080, or a Mac with 24GB unified memory.

In other words, this time 'local' really means local on an ordinary person's computer.

Sounds great, right?

But after looking at the file sizes, I found that the 17GB claim still needs a footnote... The UD-Q4_K_XL file Unsloth recommends is itself 17.9GB, and the vision projection file mmproj is about another 0.93GB. After the model loads, the system, context, and KV Cache will continue to consume memory too.

So 17GB is the floor for being able to run it; 24GB is what I consider a more reliable starting point.

Let's level the information first.

This time Qwen 3.8 has released two models of completely different scales.

Qwen3.8-27B is a 27B Dense model, natively supporting images and video, with thinking enabled by default, a native context of 262,144 tokens, and expandable to about 1M via YaRN.

It's also the version most people should download.

Qwen3.8-2.4T-A95B has 2.4T total parameters, activating 95B each time. It's a text-only, forced-thinking model, and it's still a bit of a distance from ordinary computers.

Let's figure out the 27B first.

Qwen3.8 27B really isn't just a supporting act this time

Qwen3.8-27B's parameter count hasn't become dramatically larger; the improvements mainly appear in Coding, Agent, and computer operation.

In the official table, Terminal Bench 2.1 rose from Qwen3.6-27B's 63.4 to 73.0. SWE-bench Pro rose from 53.5 to 61.7, surpassing Qwen3.7-Plus's 57.6 and also surpassing Opus4.6 Max's 53.4 in the table.

DeepSWE 1.1 is even more dramatic, rising from 13.3 to 42.2. Qwen's own QwenSWEBench rose from 49.3 to 79.0.

Qwen3.8-27B Text, Coding & Agent Benchmarks

Image source: Qwen Official Model Card

Long tasks also improved. CoWorkBench is 70.7, Qwen3.7-Plus is 65.1, and Opus4.6 Max in the table is 68.2. LiveCodeBench v6 reached 90.3, higher than Qwen3.7-Plus's 89.6 and Opus4.6 Max's 88.8.

Terminal Bench 2.1 is still higher for Opus4.6 Max at 78.2. On GPQA Diamond, Qwen3.8-27B's 89.2 is also lower than Qwen3.7-Plus's 90.3 and Opus4.6 Max's 91.3. HLE is 30.8, still a gap from Opus4.6 Max's 40.0.

So the direction of this improvement is very clear: Qwen3.8-27B has pushed a local 27B model's code execution, long task, and Agent capabilities a big step forward.

The multimodal part explains even better why the reception has been so good.

OSWorld-Verified rose from 63.9 to 84.3, WebArena-Verified from 48.8 to 64.8. AndroidWorld is 81.9, Vision2Web is 62.9, SWE-MM is 38.6.

These tests are no longer just about image recognition and Q&A. They involve looking at screens, operating computers, using browsers, and completing a software task.

Qwen3.8-27B Multimodal, Computer Use & Vision Benchmarks

Image source: Qwen Official Model Card

These numbers come from the Qwen official model card. QwenSWEBench and CoWorkBench are also Qwen's own internal evaluations; for SWE-bench Pro, except for Opus4.6 Max which uses the official score, all other models were re-run by Qwen using the Claude Code harness.

You could say, running a Qwen 3.8 27B locally gives you an Opus 4.6.

The most easily overlooked step in local deployment is this one.

When people see that 27B can be compressed to 9GB, their first reaction is that it's so small, so I can just run it casually.

Unsloth this time uses Dynamic V3.0 GGUF, currently still in Preview.

In Unsloth's published top-1 and KLD analysis, the 9GB IQ2_XXS is 83.5% smaller than the 54.7GB BF16, with capability retention at 82.5%.

This number is usable, but the loss is also obvious.

It's fine for trying out, but for complex code, long Agents, and tool calls, I wouldn't use 2-bit as the main model.

Version	Approx File Size	Currently Confirmed Quality Data	How I Would Choose
`IQ2_XXS`	~9.0GB	Unsloth measured 82.5% retention	Trying out on small memory
`UD-Q3_K_XL`	~13.4GB	Full breakdown still pending	Starting point for 16GB machines
`UD-Q4_K_XL`	~17.9GB	Full breakdown still pending	Priority for 24GB machines
NVFP4	~23.4GB	top-1 retention 92%–97%	Blackwell GPU
BF16	~54.7GB	Original precision	64GB unified memory, 80GB GPU, or multi-GPU

NVFP4's data is even more impressive.

In Unsloth's tests, it's about 1.5x faster than BF16, with top-1 retention between 92% and 97%. Code data is 96.68%, Chinese is 93.55%, chat is 92.15%.

The cost is: NVFP4 requires Blackwell. RTX 4090, 3090, and similar cards can't run it; you need at least RTX 5090, DGX Spark, B200, or B300 generation.

Unsloth provided a configuration table. The total memory here is system RAM plus VRAM, or Mac unified memory.

Quantization	Recommended Total Memory	How to Choose
2-bit	11–13GB	Can run, quality loss more noticeable
3-bit	13–16GB	16GB machines start here
4-bit	17–19GB	24GB machines prioritize this
6-bit	24GB	Need to leave space for system and context
8-bit	31GB	48GB+ more suitable
BF16	56GB	Original precision, workstation route

Quantization sounds complicated, but you just need to remember one thing: the lower the bit count, the smaller the footprint, and the quality loss is usually more noticeable.

So:

16GB machine, choose UD-Q3_K_XL, file about 13.4GB.
24GB machine, choose UD-Q4_K_XL, file about 17.9GB.
32GB machine, still recommend running Q4 first, leaving space for system and context.
48GB–64GB machine, can consider UD-Q8_K_XL, file about 31.5GB.

There's another pitfall.

256K is the upper limit the model supports, not something you must max out on first launch. The longer the context, the larger the KV Cache, and the longer it takes to read the prompt the first time.

Start uniformly from 32K the first time. Once the model, speed, and memory are all normal, then increase to 64K, 128K.

The easy way: Unsloth Desktop / Studio

If you don't want to touch compilation parameters, you can directly download Unsloth Desktop.

It supports macOS, Windows, and Linux.

Open Model hub, search for Qwen3.8-27B, then choose the quantization version according to your memory.

Unsloth will automatically handle most inference parameters and will also recognize multiple GPUs. When VRAM isn't enough, it can also offload some weights to system memory.

Thinking, Preserved Thinking, Web Search, Code Execution, and tool calling are all added in too.

This is quite important for local Agents, saving you from the situation where the model is running but the tools can't connect...

Image source: Unsloth; Dynamic 4-bit Qwen3.8-27B running in Unsloth Desktop

If you prefer a browser interface, you can also manually install Unsloth Studio.

Mac, Linux, and WSL open terminal:

curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio -p 8888

Windows uses PowerShell:

irm https://unsloth.ai/install.ps1 | iex
unsloth studio -p 8888

Then open in browser:

http://127.0.0.1:8888

First launch will ask you to create a password. Enter Model hub, search for Qwen3.8-27B-GGUF, 24GB machines choose Q4, 16GB machines choose Q3 first.

If only using on your own computer, don't add -H 0.0.0.0 to the launch command. This parameter exposes the service to other devices on the same network; definitely don't do this on the public internet.

If you want to connect your own tools, use llama.cpp

For those already using llama.cpp, first update to the latest version, then download Unsloth's GGUF from Hugging Face.

24GB machines directly download 4-bit:

pip install -U "huggingface_hub[cli]"

hf download unsloth/Qwen3.8-27B-GGUF \
  --local-dir unsloth/Qwen3.8-27B-GGUF \
  --include "*UD-Q4_K_XL*"

Then launch:

./llama.cpp/llama-cli \
  --model unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf \
  --ctx-size 32768 \
  --temp 1.0 \
  --top-p 0.95 \
  --top-k 20 \
  --min-p 0.0

16GB machines replace UD-Q4_K_XL in the download rules and model name with UD-Q3_K_XL.

When compiling llama.cpp on Mac, just turn off CUDA; Metal is enabled by default. NVIDIA uses CUDA, AMD can use HIP or Vulkan.

If you want to connect it to OpenCode, Open WebUI, or your own program, replace llama-cli with llama-server, then let the client access the local OpenAI-compatible API.

Qwen3.8 has thinking enabled by default. For complex code and Agent tasks, first use the officially recommended parameters:

temperature = 1.0
top_p = 0.95
top_k = 20
min_p = 0.0
context = start from 32768

For regular chat where you don't want it thinking for a long time, you can turn off thinking:

--chat-template-kwargs '{"enable_thinking":false}'

The recommended parameters for non-thinking mode are temperature=0.7, top_p=0.80, top_k=20, presence_penalty=1.5.

If you want to view images, remember to also download mmproj-F16.gguf from the same GGUF repository, and add --mmproj mmproj-F16.gguf when launching. Downloading only the language model file means no image capability.

Ollama and LM Studio, more suitable for those who don't want to mess with commands

Ollama has already listed the official Qwen3.8 model. Default is 27B, model about 18GB, labeled as supporting 256K context, images, tool calling, and thinking mode.

After installing Ollama, one command:

ollama run qwen3.8

Apple Silicon can also try the MLX version:

ollama run qwen3.8:27b-mlx

LM Studio's approach is also very simple.

Enter Discover, paste in Unsloth's GGUF repository address, then choose Q3, Q4, or Q8.

It's suitable for those who want to adjust VRAM offload ratios, context, and sampling parameters themselves.

When you need a local API, start the Server on the Developer page, and it can be used by other tools.

The difference between these two routes is simple: Ollama is suitable for getting running with one command, LM Studio is suitable for adjusting parameters while looking at the interface.

For team use, use vLLM and SGLang

For one person, llama.cpp is enough. If you're designing for several people calling simultaneously, vLLM and SGLang are more suitable.

Qwen officially has already provided a vLLM recipe and SGLang cookbook. They support continuous batching, Prefix Cache, multi-GPU, and OpenAI-compatible API, mainly addressing concurrency and throughput.

If you have a Blackwell GPU with at least 24GB VRAM, such as RTX 5090, DGX Spark, B200, or B300, you can use Unsloth's NVFP4 weights. The file is about 23.4GB, and Unsloth gives a speed about 1.5x faster than BF16. Current documentation requires vLLM>=0.25.0.

vLLM:

vllm serve unsloth/Qwen3.8-27B-NVFP4

If you need MTP speculative decoding, add:

vllm serve unsloth/Qwen3.8-27B-NVFP4 \
  --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

SGLang:

python -m sglang.launch_server \
  --model-path unsloth/Qwen3.8-27B-NVFP4 \
  --speculative-algorithm NEXTN \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4

Don't download NVFP4 for older NVIDIA cards. For 24GB cards like RTX 4090, 3090, continuing with GGUF + llama.cpp is more hassle-free. Official FP8 weights are about 30.9GB, more suitable for 48GB GPUs or multi-GPU servers.

Tool calling also requires enabling the corresponding parser. vLLM's Qwen3.8 recipe uses --enable-auto-tool-choice --tool-call-parser qwen3_coder.

As for 2.4T, don't consider it for personal play

Unsloth has also made Qwen3.8-2.4T-A95B into GGUF.

Official BF16 weights require 4.9TB, Q8 about 2.6TB. Unsloth's newly made Dynamic 1-bit version compressed it to 397GB, already a 91% reduction.

And then?

The smallest 397GB version still recommends preparing at least 450GB RAM. Pure CPU can start, but don't think too much about speed. Production deployment is even more夸张, TokenSpeed's reference configuration is already 16 GPUs, two 8-card nodes.

So the opening of 2.4T mainly benefits quantization teams, inference frameworks, and enterprises with servers.

Ordinary users directly running 27B actually get vision and non-thinking modes too, for a more complete experience.

So running Qwen3.8 locally really is doable.

16GB can try it out, 24GB can already use it seriously, 32GB is even less pressure.

Last time I wrote about DeepSeek-V4-Flash local deployment, it had to start from 110GB; this time Qwen3.8-27B's 4-bit is only 17.9GB.

For someone like me with a base-model 32GB Mac, this time I finally don't have to wait anymore...

Deploying immediately, will give everyone actual feedback.

Today, I have to say at least a little: Alibaba is awesome.

References: