Artificial Intelligence

A Decade-Old Server Runs Qwen 3.5 4B: Three Counterintuitive Tuning Choices

By 敖老师 · Nov 6, 2026

Read original on juejin.cn ↗ Google Translate ↗ Alt translation

A ten-year-old server can still host a small model for internal tools, but only by accepting minute-level latency in exchange for correctness: disabling flash attention and letting the CPU prefetch are the price of avoiding silent garbage output. The same trade-off applies to any unsupported GPU/backend combination where flash attention is known to be buggy.

Summary

The article documents running Qwen 3.5 4B on a ten-year-old Xeon E5-2670 v2 with an RX 580 2048SP. The hardware lacks AVX2 and ROCm support, leaving Vulkan as the only GPU path. Three decisions define the setup: turn off flash attention, let the CPU handle prefill, and lock all settings in a fixed alias. The result is a working OpenAI-compatible endpoint for LAN clients like WorkBuddy and Dify, but with minute-level latency and no room for GPU offload at larger context.

Seven parameter changes are detailed, with the most critical being OLLAMA_FLASH_ATTENTION=false. With it enabled, prompts over 1650 tokens silently produce a stream of zeros; turning it off drops GPU offload from 33/33 layers to 1/33 and generation speed from 8.6 to 5.5 tok/s, but eliminates the garbage. Other changes include raising num_ctx from 8192 to 16384 (costing ~20 extra seconds per turn), increasing num_predict to 2048 to avoid truncation, and setting OLLAMA_HOST, keep_alive, and max_loaded_models for LAN serving and memory limits.

Integration with WorkBuddy reveals that prompts are fixed at 4098 tokens with no history, so caching never helps and each turn is a full prefill plus generation. Reasoning mode must be disabled client-side via reasoning_effort: "none" on the /v1 endpoint; server-side attempts fail. The overall trade-off is predictable slowness over silent errors, and the author provides a troubleshooting table for common symptoms.

Takeaways

— Disabling flash attention on Polaris (gfx803) via Vulkan eliminates silent garbage output on prompts over 1650 tokens, but reduces GPU offload from 33/33 to 1/33 layers and generation speed from 8.6 to 5.5 tok/s.

— The RX 580 2048SP has 8GB VRAM; Qwen 3.5 4B Q4_K_M uses 3.3GB, making it the only viable model size; 8B is borderline and 14B+ drops to 1-2 tok/s with CPU mixing.

— OLLAMA_FLASH_ATTENTION=false is the critical environment variable; OLLAMA_HOST=0.0.0.0:11434 enables LAN access, OLLAMA_KEEP_ALIVE=-1 keeps the model resident, and OLLAMA_MAX_LOADED_MODELS=1 prevents VRAM overflow.

— Raising num_ctx from 8192 to 16384 increases per-turn fixed overhead from ~56s to ~76s because the GPU can't offload layers at 16384 context, falling back to near-pure CPU inference.

— num_predict was raised from 1024 to 2048 to avoid truncation that triggered a second request and a fabricated "Token overflow recovery mode" narrative from the model.

— WorkBuddy sends a fixed 4098-token prompt each turn with no history, so KV cache never hits; measured time: 56s prefill + 97s generation (544 tokens) = 3 minutes server time, UI shows 3:54.

— For OpenAI-compatible /v1/chat/completions, reasoning mode must be disabled with reasoning_effort: "none"; think:false is ignored, and server-side attempts (Modelfile PARAMETER think false, system prompt) all fail.

— With flash attention re-enabled, input prompts must stay under 1500 tokens (approximately 1000 Chinese characters), otherwise the model silently emits a long string of zeros.

— OLLAMA_NUM_PARALLEL=2 gives only 1.05x speedup over serial because a single request already saturates memory bandwidth; at 16384 context, Ollama automatically forces n_slots to 1.

Conclusions

The decision to accept 5.5 tok/s and a 1/33 layer offload rather than risk silent garbage shows that for programmatic APIs, correctness beats speed; a wrong answer without an error flag is worse than a slow one.

The 1650-token threshold for flash-attention failures is specific to this Vulkan/Polaris combination, but it demonstrates that such bugs can be characterized and worked around; the risk is that someone later flips the knob and doesn't know the threshold.

Ollama's server-side inability to disable thinking (Modelfile PARAMETER think false fails) pushes the responsibility to clients, which is a design gap for OpenAI-compatible endpoints that expect a universal way to control reasoning.

The fixed 4098-token prompt from WorkBuddy is a client design decision that negates any benefit of KV-cache reuse; this is a reminder that prompt engineering and client behavior can dominate model-level optimizations.

Concepts & terms

flash attention

An optimized attention mechanism that reduces memory usage and speeds up inference, but can be numerically buggy on certain GPU/driver combinations, here on Vulkan with Polaris.

Vulkan backend

A graphics and compute API used by Ollama on Windows for AMD cards unsupported by ROCm; it offloads model layers but has limited support for non-standard attention structures.

Q4_K_M

A 4-bit quantized model format (K-quant) that balances size and quality; for Qwen 3.5 4B it produces a 3.3 GB file.

prefill

The phase where the model processes the input prompt to generate the initial key-value cache; on a CPU-only path it can take tens of seconds.

KV cache

Key-value cache used during token generation to avoid recomputing attention for previous tokens; its size grows with context length and can double when num_ctx is raised.

Source: juejin.cn ↗ Google Translate ↗ Backup ↗