跪拜 Guibai
← Back to the summary

Running Qwen3.8-27B Locally: The Real Headache Isn't VRAM, It's a Default Setting That Turns It Into a Rambling Philosopher

On the afternoon of Thursday, August 14th, I saw the news that Qwen3.8-27B had been open-sourced. I didn't take it too seriously—just another '27B beats Claude' clickbait title, right?

But by evening, QbitAI and Almost Human were all pushing it, and it shot straight to number one on the Hugging Face trending board. I flipped through the official benchmark data: SWE-bench Pro scored 61.7, over 8 points higher than Claude Opus 4.6 Max's 53.4. QwenSWEBench was even more outrageous, 79.0 versus 63.8, a 15-point lead.

This couldn't just be dismissed as 'marketing benchmarks.'

Fine, I had nothing to do over the weekend anyway, so I downloaded it to give it a try.

First, solve the 'can I even run it' problem

A 27B-parameter dense model requires 55GB of VRAM at BF16 full precision. My RTX 3090 only has 24GB, so forget about it.

Fortunately, the Unsloth community released a GGUF quantized version on the same day. There are several quantization levels; I went straight for Q4_K_M—17GB, which fits perfectly into 24GB of VRAM, leaving 7GB for the KV Cache.

A note for those with less VRAM: Q4_K_M is the sweet spot for 24GB (both 3090/4090 work). 16GB cards need to use Q3 quantization, which degrades performance. For 12GB and below—honestly, running an 8B model is more appropriate; don't force it.

I used llama.cpp as the inference engine, but this is where I hit the first pitfall.

The llama.cpp version must be the latest, or it will error out immediately

Qwen3.8 does not use a pure Transformer architecture—out of 64 layers, 48 are DeltaNet linear attention, and only 16 are traditional full attention. This 3:1 hybrid architecture is the reason for its fast inference speed (linear attention has constant computation cost that doesn't grow with context length), but it also means older inference engines don't recognize this structure at all.

The first time I tried, I used a llama.cpp compiled last week, and loading the model threw an error immediately:

unknown architecture: qwen3.8-hybrid

After searching around, I found that you must use version b10451 or higher for support. Recompiled:

git pull
make -j8

This time it loaded successfully. If you encounter a similar architecture recognition error, your first reaction should be to update the engine version, not to doubt whether the model file downloaded correctly.

The default settings are a huge pitfall: the model turns into a 'rambling philosopher'

The model was running, but what came next was what really wasted over an hour of my time.

Qwen3.8-27B adds a new reasoning_effort parameter that controls the model's depth of thought. Three levels:

The problem is—the default value is xhigh.

What does this mean? You ask it 'what is 1+1,' and it can give you a 2000-word reasoning process. Simon Willison (yes, the co-creator of Django) documented a classic test on his blog: asking the model to draw an SVG of a 'pelican riding a bicycle.'

The result: the model spent 21 minutes, burned through 22,276 reasoning tokens, and finally drew something barely recognizable.

I reproduced a similar situation locally. I asked it a very simple Python list deduplication question, and it first spent over 3000 tokens 'thinking' about whether to use set, whether to preserve order, whether to handle nested lists... and finally gave a 40-line answer, 30 lines of which were edge-case handling it added itself.

After switching reasoning_effort to medium, the whole world went quiet. The same question was handled within 500 tokens, and the speed jumped directly from 3 tokens/s to 38 tokens/s.

So if you deploy it and find the model 'reacts slowly' or 'answers are absurdly verbose,' don't rush to doubt the quantization precision; check this parameter first.

How to set it in llama.cpp:

llama-server \
  -m Qwen3.8-27B-Q4_K_M.gguf \
  -ngl 99 -fa 1 \
  -c 65536 \
  --temp 0.7

Then add a line in the system prompt:

reasoning_effort: medium

Or if you're calling it via an OpenAI-compatible API format, pass it in the request body:

{
  "messages": [...],
  "reasoning_effort": "medium"
}

Performance data: much faster than expected

After switching to medium, I ran it all afternoon and recorded the speed data (room temperature 26°C, GPU power limit at default, untouched):

Scenario Speed Feel
Prompt processing (feeding long text) Up to 1385 tokens/s A 2000-word article read in 1 second
Text generation About 38 tokens/s 3-4 times faster than normal reading speed
With image input About 25 tokens/s Slightly slower but completely usable

What does 38 tokens/s feel like? When you open the GPT-4o web version, the perceived speed is about the same. The difference is that this runs entirely on my own graphics card, no internet needed, no subscription fees.

Also worth mentioning is Multi-Token Prediction (MTP). Qwen3.8 has this capability built-in; in llama.cpp, enable it with --spec-type draft-mtp. After I turned it on, generation speed could increase by another 20-30%, but VRAM usage goes up by about 2.5GB. A 24GB card just barely handles it; 16GB users are advised to turn it off.

But I digress. In between, I had it write a regex—parsing those UA strings in Nginx logs with all kinds of escape characters. It got it right in one shot, even cleaner than the version I pieced together after searching StackOverflow for half an hour last time. This kind of thing was simply impossible for local small models before.

Initial capability probe: the coding chops are real

After sorting out the speed, I tried a few scenarios.

Coding ability: Gave it a real requirement—migrating an Express project from CommonJS to ESM. It accurately identified all the require statements that needed changing, including how to handle dynamic imports, and even swapped out details like __dirname being unavailable in ESM for the import.meta.url approach. No screw-ups.

Image understanding: I threw a screenshot of a terminal error at it, and it accurately identified the error message and gave a fix. But there were also failures—I took a screenshot of code with Chinese comments, and it recognized '这里要加锁' (need to lock here) as '这里要加鏁' (need to add a mythical beast here). I momentarily suspected the quantization had corrupted the Chinese vocabulary. Later, switching to English screenshots worked perfectly, which was puzzling.

Long text: Tried 65K context (the maximum stably supported by llama.cpp currently), fed it a 200,000-word novel and asked it to sort out character relationships. It ran for about 40 seconds, and the results were basically accurate; the main characters' relationships were all mapped out. The official specs say it natively supports 262K, but llama.cpp hasn't fully adapted to it yet; we'll have to wait for subsequent updates.

There are a few more pitfalls to mention.

Ollama doesn't have official support yet. I initially tried to be lazy and run ollama run qwen3.8:27b to get it done in one step, only to find the model wasn't in the list yet. To run it, you currently have to go through llama.cpp or LM Studio (the latter's backend is also llama.cpp, but it wraps a GUI for you).

Don't rush for 131K+ context. Although the model natively supports 262K, and YaRN technology can extend it to 1M, the adaptation for these long contexts in llama.cpp and SGLang is still in progress. Currently, 65K is the stable usable upper limit.

A BF16 dual-GPU setup is not feasible. If you have a heterogeneous dual-GPU setup like a 4090+4070Ti, don't think about using tensor parallelism to run full precision—vLLM doesn't support heterogeneous VRAM pools, and even splitting will cause the smaller card to OOM. Running the official FP8 quantized version on a single 4090 is the most realistic plan.

Is it ultimately worth the trouble

To be honest: the official benchmark data showing it 'surpasses Claude Opus 4.6 Max' definitely takes a hit on the Q4 quantized version. But even at a 20-30% discount, a locally free 27B model that can write code, understand images, and read long texts—what more could you ask for.

I've already set it to auto-start on boot anyway. When llama.cpp later adapts to 262K long context, I'll come back and fill in this gap.