A Decade-Old Server Runs Qwen 3.5 4B: A Tuning Log Forced by Hardware

Let me state the conclusion first: a ten-year-old machine can run a model and serve it externally, but the process isn't "install and done". This generation of hardware (AMD Polaris GPU + an Ivy Bridge CPU without AVX2) forced three counterintuitive decisions: turn off flash attention, accept the CPU as the main compute, and pin parameters down with a fixed alias.

Environment: Intel Xeon E5-2670 v2 / 32 GB DDR3L / AMD RX 580 2048SP 8 GB / Windows 10 Pro 19045, running Ollama 0.40.14 + Qwen 3.5 4B (Q4_K_M), providing an OpenAI-compatible API to LAN platforms like WorkBuddy and Dify.

1. Hardware Directly Defines the Solution Boundaries

Item	Actual Configuration	Impact on Solution
CPU	Xeon E5-2670 v2, 10 cores / 20 threads, Ivy Bridge, AVX only, no AVX2	Can only act as fallback compute; prefill about 54~74 tok/s
Memory	32 GB DDR3L-1600	Handles the 9.6 GB model + KV usage
GPU	RX 580 2048SP / 8 GB, Polaris gfx803	ROCm not supported (only RDNA3 from RX 7600 onward), only Vulkan works
OS	Windows 10 19045	Ollama 0.40.x enables Vulkan backend by default, so Polaris is usable

Three hard boundaries:

1. No ROCm, only Vulkan. Vulkan on Polaris has incomplete support for "non-standard attention structures". This is the root cause of all subsequent trade-offs.
2. 8 GB VRAM sets the model limit. A 4B Q4 (3.3 GB) is comfortable, an 8B Q4 (about 5 GB) is borderline, and anything above 14B requires CPU mixing, dropping speed to 1~2 tok/s.
3. Model selection is locked in. The only usable model is Qwen3.5 4B (hybrid architecture). Switching to a pure dense model (like qwen3:4b-instruct-2507) offloads better to the GPU, but it randomly outputs long strings of zeros in silent failure, so it was abandoned and deleted.

2. The Final Configuration

Externally there is only one name: qwen-api. All platforms use this model name; when swapping the underlying model, only the alias is rebuilt, and caller code stays unchanged.

FROM qwen3.5:4b-q4_K_M

PARAMETER num_ctx 16384
PARAMETER temperature 0.7
PARAMETER top_p 0.9
PARAMETER repeat_penalty 1.05
PARAMETER num_predict 2048

SYSTEM "You are the local model assistant running on this Windows PC. Answer concisely and accurately, default to Chinese; if uncertain, say so clearly, don't fabricate facts."

User-level environment variables (Ollama reads them only at startup; the service must be restarted after changes):

Variable	Value	Effect
OLLAMA_HOST	0.0.0.0:11434	Listens on all network adapters, otherwise other LAN machines can't connect
OLLAMA_KEEP_ALIVE	-1	Model permanently resident, never actively released
OLLAMA_MAX_LOADED_MODELS	1	8 GB VRAM only fits one model
OLLAMA_NUM_PARALLEL	2	Nominal 2-way concurrency, measured but forced to 1 slot
OLLAMA_FLASH_ATTENTION	false	Key reverse setting
OLLAMA_CONTEXT_LENGTH	16384	Fallback value; actual value is num_ctx in the Modelfile

The service is launched via a startup script (set variables first, then start), avoiding the common pitfall of "environment variables changed but didn't take effect". No auto-start on boot; manually run it after restart.

3. Seven Parameter Adjustments, Why Each Was Made

3.1 Flash Attention: From On to Off (The Most Important Decision in This Deployment)

With it enabled by default it looks perfect: 33/33 layers on GPU, 100% GPU, 3.4 GB usage, 8.6 tok/s generation.

But real-world tests showed that long prompts silently output garbage (a long string of zeros), and it reproduced consistently:

Prompt tokens	Output
1633	Normal
1708	Garbage (all zeros)
1783	Garbage
1858	Garbage (reproduced twice in a row)

After turning off FA, the same 1858-token prompt worked completely normally. The root cause is a numerical defect in flash attention on the Vulkan backend with Polaris for long sequences. It's not a corrupted model or misconfigured parameters; tuning cannot fix it.

The cost, stated honestly:

	FA on	FA off (current)
Layers on GPU	33/33	1/33 (0/33 at 16384 context)
Usage	3.4 GB	8.7 → 9.6 GB
Generation speed	8.6 tok/s	5.5 tok/s
Prefill	55 tok/s	72~74 tok/s (at 8192)
Long prompts >1650 tokens	Garbage output	Normal

The trade-off logic in one sentence: predictable slowness is better than unpredictable errors. For a programmatic API, when the body suddenly turns into a string of zeros without an error, that's far more dangerous than being slow.

3.2 num_ctx: 8192 → 16384

Initially set to 8192, "good enough". Later, to connect multi-platform clients like Dify that accumulate conversation history, it was raised to 16384. For the same 4098-token prompt, the cost:

Config	Prefill	Fixed per-turn overhead	Usage	Offload
8192	5556 s (7274 tok/s)	about 56 s	8.7 GB	1/33 layers
16384 (current)	about 76 s (54 tok/s)	about 76 s	9.6 GB	0/33 layers

Each turn costs about 20 seconds more because the KV cache doubles and the GPU can't hold the layers, degrading to near-pure CPU inference.

But the scenario matters: WorkBuddy has a fixed 4098-token prompt per turn with no history. 4098 + 2048 = 6146, which is already within 8192. For WorkBuddy alone, 8192 is cheaper; multi-platform sharing is why 16384 is needed.

3.3 num_predict: 1024 → 2048

The first time WorkBuddy was integrated, it hit a wall: when the answer reached 1024 tokens it hit the limit and got truncated, the client automatically sent a second request to continue, and the two parts together took 7 minutes 38 seconds. During the continuation, the model even fabricated a "Token overflow recovery mode" narrative.

After raising it to 2048 this disappeared, at the cost of doubling the worst-case time for long answers.

3.4 OLLAMA_HOST: 127.0.0.1 → 0.0.0.0

The default only listens on localhost, so WorkBuddy on another computer couldn't connect. Changing it to 0.0.0.0 made it listen on :::11434, and LAN access to /api/version returned 0.40.14.

Security note: this means any device on the same subnet can call the GPU without authentication. Enable it only on a trusted network; to tighten, bind to a specific NIC IP and restrict source IPs.

3.5 keep_alive=-1 and max_loaded=1

This machine is only for running models, so the model is kept permanently in VRAM (ps shows UNTIL as Forever), avoiding the ~20-second load time on every request. Also, limit to only one loaded model to prevent the 8 GB VRAM from being blown by a second model.

3.6 NUM_PARALLEL=2: Kept But No Benefit

Measured 2-way concurrency: single path 8.7 s, two paths 8.3 s / 16.6 s, total wall clock 16.6 s, vs serial 17.4 s, only 1.05x. A single request already saturates memory bandwidth; adding concurrency doesn't increase throughput, only queues.

More crucially, after context was raised to 16384, logs show n_slots = 1, Ollama itself forced concurrency back to 1.

3.7 Fixed Alias Pins Parameters

Pinning context, sampling, output limits, and system prompt all into the qwen-api Modelfile means any client that connects automatically inherits the correct parameters, no need to configure each platform separately.

4. What Connecting a Third-Party Agent Client Revealed

WorkBuddy for example, the configuration is simple:

Base URL is http://local-IP:11434/v1 (must include /v1), Key can be any non-empty string, model name must match ollama list, and thinking mode is unchecked.

Three facts surfaced after wiring it up, more worth remembering than the model itself:

1. Every turn sends a fixed 4098-token prompt, recomputed from scratch. The log shows four consecutive requests all at 4098, not a single token off, because it doesn't carry conversation history, so the cache can never hit. That's why "you always wait a minute no matter what question", unrelated to question length. Incidentally, it also explains why it doesn't remember the previous turn, not because the model is forgetful.
2. Disabling "tool calls" doesn't shorten the prompt. Even with it unchecked, it's still 4098; the bulk is the client's own injected project memory and system prompt.
3. Real time breakdown for one turn: prompt 4098 tokens (prefill 56 s) + generation 544 tokens (97 s) = 3 minutes server time, UI shows 3 minutes 54 seconds; the extra time is the client's own environment startup.

Screenshot showing a one-line explanation of the cache, 544 tokens generated, complete body, no residue from reasoning:

Simple arithmetic also verified accuracy and timing (2+3*4 returns 14, 9 to the third power returns 729):

5. Two Other Pitfalls

5.1 Thinking Mode: Two Interfaces, Two Ways to Turn Off, Server-Side Can't

Qwen3.5 defaults to thinking enabled, which is a disaster for programmatic calls: a one-line answer burns 500+ tokens, and the body may be empty.

Interface	How to disable
Native /api/chat, /api/generate	"think": false
OpenAI-compatible /v1/chat/completions	"reasoning_effort": "none" (think:false has no effect here)

Measured with the same question and a 150-token limit:

Parameter passed	Time	Tokens	Body
Nothing	26.2 s	150 (burned full)	Empty
think: false (on /v1)	25.9 s	150	Empty (parameter ignored)
reasoning_effort: "none"	6.2 s	34	65 chars, normal

Three server-side attempts to pin it down all failed: PARAMETER think false in Modelfile (reports unknown parameter), a system prompt forbidding thinking, and adding /no_think. Conclusion: thinking can only be disabled on the caller side.

The exception is WorkBuddy: when "thinking mode" is unchecked in its UI, output is clean body text, meaning it handles it internally.

5.2 The 1650-Token Red Line

FA is off in the current config; this red line only triggers when FA is on. With the current config, long prompts are fine. But if someone re-enables FA for speed, they must also constrain every input to no more than 1500 tokens (about 1000 Chinese characters), otherwise the model silently outputs garbage.

6. Maintenance and Troubleshooting

& "$env:LOCALAPPDATA\Programs\Ollama\ollama.exe" ps
& "$env:LOCALAPPDATA\Programs\Ollama\ollama.exe" list
Invoke-RestMethod -Uri "http://127.0.0.1:11434/api/version"

To change parameters: edit Modelfile → create qwen-api -f Modelfile → takes effect on next request.

For logs, check server.log; look at prompt eval time (actual per-turn prompt length and speed), offloaded N/M layers (layers on GPU), and flash_attn (FA status).

Symptom	Check first
Empty answer	Thinking mode not off (use reasoning_effort: none on /v1)
Output a long string of zeros	Prompt too long and FA enabled (>1650 tokens)
Slow, minutes	Prefill is on CPU, check PROCESSOR usage in ps
Other computers can't connect	OLLAMA_HOST not effective or firewall blocks it
Environment variable changes not reflected	Not using startup script; service inherited old environment

Rollback: stop process → unins000.exe /VERYSILENT → delete .ollama directory → clear the 6 OLLAMA_* variables.

7. Who This Setup Is For

Suitable for: single machine, few clients, able to accept "one-minute-scale" response times, and requiring correct results without garbage.

Not suitable for: high concurrency, sub-second response times, or scenarios that need the GPU fully utilized. This hardware generation (Polaris + Ivy Bridge) has its ceiling here; the software layer is fully tuned. Further speedup requires either switching to an Nvidia card for CUDA or to RDNA3 for ROCm.