Temperature Doesn't Create Creativity — It Just Flattens the Odds
Misunderstanding temperature leads teams to crank it up expecting smarter output or down expecting zero hallucinations, when it controls neither. Getting the sampling parameters right is what keeps an agent from calling the wrong tool and a RAG pipeline from improvising over missing documents.
Every token a large language model generates is drawn from a probability distribution over its vocabulary. Temperature reshapes that distribution before sampling — low values sharpen it toward the most likely token, high values flatten it so long-shot candidates get drawn more often. Top-K then limits the pool to a fixed number of high-probability candidates, while Top-P (nucleus sampling) keeps the smallest set whose cumulative probability hits a threshold, adapting the pool size to how confident the model is.
These three knobs are the reason the same prompt can return different code, different facts, or different tool-call decisions. In agent and RAG pipelines, where deterministic tool selection and faithful retrieval matter more than variety, a temperature near zero is standard — not because it makes the model smarter, but because it suppresses sampling variance. The DeepSeek API docs even recommend tuning only one of temperature or top_p at a time, since both alter the sampling space and make it impossible to attribute output changes.
A practical tuning table maps tasks to starting ranges: 0–0.2 for tool calls and structured extraction, 0–0.3 for RAG Q&A, 0.2–0.6 for technical dialogue, 0.6–0.9 for headlines and copy, and 0.8–1.2 for story brainstorming. The core advice is to fix the prompt and test set, vary one parameter at a time, and measure accuracy, format stability, and diversity — not to copy a universal value.
Temperature is widely misunderstood as a creativity dial, but it's purely a sampling-distribution shaper — the model's knowledge and reasoning are frozen before sampling begins.
The advice to tune only temperature or top_p, not both, is a practical guardrail against the common reflex of twisting every knob at once and then being unable to reproduce or debug behavior.
Calling temperature 0 'deterministic' is misleading; server-side infrastructure, floating-point non-determinism, and context drift still produce variation, so pipelines that require bit-identical output need a different strategy.
The task-based temperature table is a useful starting heuristic, but its real value is the methodology: fix the prompt and test harness, vary one parameter, and let business metrics — not intuition — pick the value.