跪拜 Guibai
← Back to the summary

Temperature Doesn't Create Creativity — It Just Flattens the Odds

Asking a large model the same question several times in a row often yields answers that aren't exactly the same.

Sometimes only the wording changes; other times, even the conclusions and code implementations differ. Many people simply attribute this to "AI randomness" and then start repeatedly adjusting temperature: set it to 0 for writing code, crank it up to 1 for writing copy.

But what exactly does temperature change? Does it directly add creativity to the model? What are Top-K and Top-P controlling?

To understand these parameters, you have to go back to the most fundamental way a large model works: predicting the next Token based on the current context.

1. A Large Model Doesn't Think Up a Complete Answer All at Once

Suppose the context already contains "你好" (hello), and the model is preparing to generate the next Token. After the Transformer computation, it doesn't immediately get a single answer but rather a set of scores for candidate Tokens.

For easier understanding, the scores can be converted into a probability distribution like the one below:

Candidate Token Probability
吗 0.60
啊 0.15
呀 0.10
美 0.05
坏 0.01
Other Tokens 0.09

"吗" has the highest probability, but the highest probability doesn't mean the model must choose it every time.

There are generally two types of strategies during the generation phase:

If greedy selection is always used, the output is usually more stable but also prone to being mechanical and repetitive; if sampling among candidate Tokens is allowed, expression becomes more diverse, but the risk of errors and going off-topic also increases.

temperature, Top-K, and Top-P adjust the candidate range and probability distribution right before the Token is actually drawn.

2. Temperature Changes the Shape of the Probability Distribution

What the model initially outputs is usually not probabilities but a set of raw scores called Logits. Temperature participates in the calculation before Softmax:

Adjusted probability = softmax(logits / temperature)

It doesn't magically add new candidate words; it readjusts the probability gaps between existing candidates.

Lower Temperature

When Temperature is less than 1, high-scoring candidates become more prominent, and the chances of low-scoring candidates drop further. The probability distribution becomes "sharper."

Original distribution: 吗 0.60, 啊 0.15, 呀 0.10...
Lowering temperature: 吗's advantage continues to expand, other candidates become harder to select

The output typically has these characteristics:

Higher Temperature

When Temperature is greater than 1, the probability gaps between different candidates shrink, and Tokens with originally lower probabilities get more opportunities. The probability distribution becomes "flatter."

The output is usually more diverse, but the following can also occur:

Therefore, what Temperature controls is not "how smart the model is," but rather how willing it is to try low-probability candidates during generation.

How to Understand Temperature of 0

Mathematically, Temperature cannot be directly divided by 0. temperature: 0 in an API is usually a special configuration provided by the server-side to try to use the most stable generation strategy possible.

It also doesn't mean you can get completely identical results under any circumstances. Model versions, server-side implementations, parallel computation, tool return content, and context changes can all cause output differences.

So a more accurate statement is:

temperature: 0 tries to minimize sampling randomness as much as possible, but it does not equal absolute determinism.

3. Top-K: Limit the Number of Candidates First

Temperature adjusts probability gaps; Top-K controls the maximum number of candidate Tokens to keep.

If you set:

Top-K = 3

The model first sorts by probability from high to low, keeping only the top three candidates:

Candidate Token Original Probability
吗 0.60
啊 0.15
呀 0.10

"美," "坏," and other candidates are excluded. The system then re-normalizes the remaining three Tokens before performing sampling.

When Top-K is small:

When Top-K is large:

Note that Top-K is not a parameter directly exposed by all large model APIs. It is common in Hugging Face Transformers and some open-source model inference frameworks; OpenAI-compatible interfaces more commonly feature temperature and top_p.

4. Top-P: Dynamically Select Candidates Based on Cumulative Probability

Top-P is also called Nucleus Sampling.

It doesn't fix the number of candidates but starts accumulating from the highest-probability Token, keeping the smallest set of candidates whose cumulative probability reaches or exceeds a specified threshold.

Suppose:

Top-P = 0.8

Start accumulating according to the previous probability distribution:

吗: 0.60
吗 + 啊: 0.75
吗 + 啊 + 呀: 0.85

After adding the third Token, the cumulative probability exceeds 0.8, so this time the three candidates "吗, 啊, 呀" are kept.

The difference between Top-P and Top-K lies in whether the number of candidates is fixed:

Parameter Selection Method Characteristic
Top-K Fixed retention of the K highest-probability Tokens Candidate count is fixed
Top-P Retains the smallest set reaching cumulative probability P Candidate count varies with the distribution

When the model is very certain, the cumulative probability of a few Tokens can reach Top-P; when probabilities are more dispersed, more Tokens need to be kept. Therefore, Top-P is more adaptable to different contexts.

5. How the Three Parameters Participate Together in Generation

Conceptually, a single Token sampling can be understood as:

Context
→ Model calculates Logits for the next Token
→ Temperature adjusts probability gaps
→ Top-K or Top-P narrows the candidate range
→ Re-normalize probabilities
→ Draw one Token according to probability
→ Add the new Token to the context
→ Continue predicting the next Token

This process loops until the model generates an end token or reaches the maximum output length.

However, more parameters don't mean more precise control. Taking the current DeepSeek Chat Completion interface as an example, the official documentation suggests usually modifying either temperature or top_p, not both simultaneously.

The reason is simple: both parameters change the final sampling space. If adjusted at the same time, it's hard to tell which parameter caused the output change, which is also not conducive to subsequent testing and reproduction.

The correct parameter tuning approach should be:

  1. Keep the Prompt, model, and test data unchanged;
  2. Change only one parameter at a time;
  3. Execute repeatedly using the same set of questions;
  4. Compare accuracy, format stability, repeatability, and content diversity;
  5. Choose parameters based on real business results, rather than directly copying a "universal value."

6. Why Agent and RAG Projects Often Use temperature: 0

In an existing LangChain Agent project, the model configuration uses:

const model = new ChatOpenAI({
  modelName: process.env.DEEPSEEK_MODEL || 'deepseek-chat',
  apiKey: process.env.DEEPSEEK_API_KEY,
  temperature: 0,
  configuration: {
    baseURL: 'https://api.deepseek.com/v1',
  },
});

A low Temperature is used here not because Agents don't need creativity at all, but because such tasks prioritize controllability.

For example, an Agent needs to decide:

If the model overly pursues diversity in these steps, it might choose the wrong tool, generate parameters that don't conform to the Schema, or take very different execution paths under the same input.

RAG has similar needs. The retrieved materials already provide a factual basis; the model's main task is to organize an answer based on the materials, not to improvise freely. Therefore, a lower Temperature is generally more suitable for Q&A, summarization, and fact-organization tasks.

But special attention must be paid:

Lowering Temperature can only reduce output fluctuation; it cannot fix incorrect knowledge, retrieval failures, or missing context.

If RAG does not retrieve the correct documents, even with temperature: 0, the model may still generate an answer based on erroneous or incomplete context.

7. Temperature's Position in a LangChain Workflow

LangChain's role is to connect nodes like Prompts, models, output parsers, and tools into a workflow.

For example, a basic chain can be represented as:

const creativeChain = storyPrompt
  .pipe(creativeModel)
  .pipe(outputParser);

await creativeChain.invoke('a piece of prompt');

The corresponding execution order is:

Input
→ PromptTemplate organizes the prompt
→ ChatModel generates content based on parameters
→ StringOutputParser parses the model output
→ Returns the business result

Temperature belongs to the model node's configuration, not to PromptTemplate, nor is it a capability of the output parser.

The same application can choose different model nodes based on the task:

This design is more reasonable than using the same Temperature across the entire application, because each node in a real workflow has different requirements for stability and diversity.

8. What Ranges Different Tasks Can Start Testing From

The values below are not fixed standards but empirical ranges convenient for starting tests. Actual performance may vary with different models and service providers.

Task Temperature to Try Main Goal
Tool calls, structured extraction 0~0.2 Stable parameters, accurate format
RAG Q&A, material summarization 0~0.3 Faithful to context, reduce improvisation
Technical explanation, general dialogue 0.2~0.6 Balance between stability and naturalness
Titles and copywriting proposals 0.6~0.9 Increase expression variation
Stories, creative brainstorming 0.8~1.2 Expand the candidate space

Before raising Temperature, you should first clearly write out the Prompt's goal, context, and output requirements. A vague Prompt paired with a high Temperature will only amplify uncertainty and will not automatically yield high-quality creativity.

9. Several Common Misconceptions

1. The higher the Temperature, the smarter the model

False. Temperature does not change the knowledge the model has already learned, nor does it enhance reasoning ability; it only changes the sampling distribution during the generation phase.

2. The higher the Temperature, the more creative the content must be

False. More random only means a wider candidate range, which can also produce irrelevant, low-quality, or factually incorrect content. Creative quality also depends on model capability, the Prompt, context, and the filtering process.

3. Setting Temperature to 0 prevents hallucinations

False. Hallucinations can come from limitations in training knowledge, vague questions, missing context, retrieval errors, or tool result errors. Low Temperature only makes the model more inclined towards high-probability expressions.

4. All models support Top-K

False. Different providers expose different sampling parameters. Before writing code, you should check the current model's interface documentation; don't directly transplant parameters from one inference framework to another API.

5. Drastically adjusting Temperature and Top-P together is more effective

Not necessarily. Changing multiple variables simultaneously makes results harder to analyze. Prioritize fixing one parameter and only testing the impact of the other.

Summary

Every step of generation in a large model is essentially making a choice within the probability distribution of candidate Tokens.

A truly reliable AI application is not about finding one Temperature suitable for all tasks, but about controlling stability and diversity separately according to the responsibilities of each node in the workflow.