Qwen3.8-27B Lands on a Single GPU: Ollama, MTP, and Agent Wiring

Qwen3.8-27B is a multimodal dense model open-sourced by Qwen, released on 2026-08-14.

Qwen3.8-Flash-Next is a multimodal MoE model open-sourced by Qwen, also a preview of Qwen4, released on 2026-08-26.

I originally wanted to see if I could deploy Flash, but it won't run without a small cluster 😥 — even Q4 quantization requires over a hundred GB of VRAM.

So, I chose to deploy Qwen3.8-27B to play with — 256K context, Q4 takes about 18 GB VRAM. Consumer-grade GPUs can deploy it.

Qwen3.8-27B: https://huggingface.co/Qwen/Qwen3.8-27B
Qwen3.8-Flash-Next: https://huggingface.co/Qwen/Qwen3.8-Flash-Next

How to Deploy

There are many deployment methods, as follows:

SGLang: https://docs.sglang.io/cookbook/autoregressive/Qwen/Qwen3.8-27B
vLLM: https://recipes.vllm.ai/Qwen/Qwen3.8-27B
llama.cpp: https://llama.app/models/qwen-3-8
Ollama: https://ollama.com/library/qwen3.8

For production environments or multi-card throughput, SGLang or vLLM is recommended. For personal single-card or local use, llama.cpp is fine.

I found that Ollama has very complete quantized model versions, so I chose it for deployment here.

Ollama Deployment

Install and deploy,

# Download https://ollama.com/download
curl -fsSL https://ollama.com/install.sh | sh

# Verify
$ ollama -v
ollama version is 0.33.1

# Pull, specifying the version
#  Q4_K_M quantization ~18 GB
ollama pull qwen3.8:27b-q4_K_M
ollama pull qwen3.8:27b-mtp-q4_K_M  # Multi-Token Prediction
#  NVFP4 quantization ~18 GB (MLX)
#   Suitable for RTX 5090 (leverages fifth-gen Blackwell FP4 cores for acceleration)
#   But requires MLX backend, not yet mature on Linux, waiting for its integration update
ollama pull qwen3.8:27b-nvfp4

# Run chat, specifying the version
ollama run qwen3.8:27b-mtp-q4_K_M

# Stop service (Linux)
sudo systemctl stop ollama

Test OpenAI-compatible API,

curl -X POST http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
    "model": "qwen3.8:27b-mtp-q4_K_M",
    "messages": [
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "请用一句话解释什么是量子计算"}
    ],
    "max_tokens": 100,
    "temperature": 0.3
}'

# --- Response below ---
{"id":"chatcmpl-199","object":"chat.completion","created":1787895128,"model":"qwen3.8:27b-mtp-q4_K_M","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"","reasoning":"用户要求用一句话解释什么是量子计算。需要简洁、准确、易懂。\n\n量子计算的核心：利用量子比特（qubit）的叠加态和纠缠等量子力学特性，进行信息处理，从而在某些问题上比经典计算机具有指数级加速优势。\n\n一句话要涵盖：\n- 利用量子力学原理（叠加、纠缠）\n- 用量子比特代替经典比特\n- 能解决某些经典计算机难以处理的问题\n\n我尽量用一句通顺的话表达"},"finish_reason":"length"}],"usage":{"prompt_tokens":27,"completion_tokens":100,"total_tokens":127}}

Test image understanding (also using OpenAI-compatible API),

# Download image
curl -L -o test.jpg "https://upload.wikimedia.org/wikipedia/commons/3/3a/Cat03.jpg"
# Encode image (generate Base64)
IMG=$(base64 -w 0 < test.jpg 2>/dev/null || base64 < test.jpg | tr -d '\n')
# Prepare request
cat > request.json <<EOF
{
    "model": "qwen3.8:27b-mtp-q4_K_M",
    "messages": [
        {
            "role": "user",
            "content": [
                {"type": "text", "text": "请描述这张图片里的内容"},
                {"type": "image_url", "image_url": {"url": "data:image/jpeg;base64,${IMG}"}}
            ]
        }
    ],
    "max_tokens": 200,
    "temperature": 0.5
}
EOF
# Test request
curl -X POST http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d @request.json

# --- Response below ---
{"id":"chatcmpl-393","object":"chat.completion","created":1787895371,"model":"qwen3.8:27b-mtp-q4_K_M","system_fingerprint":"fp_ollama","choices":[{"index":0,"message":{"role":"assistant","content":"这张图片是一只橘色（姜黄色）虎斑猫的特写照片。\n\n主要内容包括：\n\n- **主体**：一只橘猫，毛色为橙黄底色带有淡淡的虎斑条纹，胸前和下巴处毛色偏白。\n- **面部细节**：它有一双黄绿色的眼睛，瞳孔呈竖线状，眼神专注地望向镜头侧前方；鼻子是粉褐色的；嘴边伸出许多细长的白色胡须，向两侧和下方散开；两只耳朵竖立，耳内可见浅色绒毛。\n- **姿态**：猫的身体朝向画面左下方，头部抬起，神情警觉而平静。\n- **背景**：背景被虚化处理，呈浅","reasoning":"用户要求描述图片内容。我需要观察图片细节：一只橘色虎斑猫的特写，绿色眼睛，粉色鼻子，白色胡须，背景虚化，有红色线条状物体。直接组织成自然的中文描述即可，不需要冗长铺垫。"},"finish_reason":"length"}],"usage":{"prompt_tokens":2523,"completion_tokens":200,"total_tokens":2723}}

Agent Usage

Claude Code

Anthropic-compatible API see: https://docs.ollama.com/api/anthropic-compatibility

export ANTHROPIC_AUTH_TOKEN=ollama
export ANTHROPIC_BASE_URL=http://localhost:11434

You can set the above environment variables, or use CC Switch with a custom config.

DeepSeek Harness

dsh model configuration: https://deepseek-harness.github.io/deepseek-harness/guide/providers

Open Settings → Models → Add Custom Provider,

Select the qwen model in the session input box,

Conclusion

Let's Go Coding ~