Temperature and Top-K Are Not Mysteries: A Practical Guide to Controlling LLM Output Randomness

Foreword

With the same prompt, a large language model returns different results each time. Many developers wonder: how exactly does the model "speak randomly"? The core mechanism behind it is word probability sampling. Two parameters, Temperature and Top-K, directly control the generation style: writing code or contracts demands rigor and correctness; prose or scripts need creativity and inspiration. If the parameters are poorly matched, the output becomes either monotonously uniform or wildly hallucinatory.

This article first explains the underlying sampling principles, then demonstrates two model pipelines through complete LangChain code: a creative generation pipeline and a rigorous, realistic pipeline, helping you controllably harness the randomness of large models in your business.

1. How Do Large Models Generate Text? The Core: Predicting the Next Word

Large model generation is a form of autoregressive text generation: given the preceding text, the model calculates a probability distribution over all candidate words in its vocabulary, selects the next word from it, and repeats this loop until generation ends.

A simple example: given the preceding text "你好" (Hello), the model outputs a probability distribution:

• 吗: 0.6
• 啊: 0.15
• 呀: 0.1
• 美: 0.05
• 坏: 0.01

Looking purely at probabilities, the highest weight is 吗. By default, always picking the highest score would make the output permanently fixed and completely uncreative. To give the model diversity, random sampling is introduced, and the two most important switches for controlling randomness are: Temperature and Top-K.

2. In-Depth Analysis of the Two Core Parameters

1. Temperature: Adjusting the Degree of Random Divergence

The value range is generally 0～1

• Lower temperature (approaching 0): Amplifies the weight of high-probability words and suppresses low-scoring ones. The model prioritizes the most reasonable, safest words, producing stable, highly repetitive output that is unlikely to go off-script. Suitable for code writing, legal contracts, data analysis, and professional Q&A.
• Higher temperature (approaching 1): Flattens the probability gaps between words, giving low-probability, uncommon words a chance to be selected. The text becomes more creative and expressive, but is more prone to factual errors and hallucinations.

Key reminder: Simply cranking up Temperature to boost creativity carries a huge risk. The model might pick words from extremely low-probability candidates, fabricating content out of thin air and dramatically increasing the hallucination rate.

2. Top-K: Defining the Candidate Word Selection Range

Top-K logic: The model first sorts all candidate words by predicted probability from high to low, retains only the top K words with the highest probabilities, and samples exclusively within this small set, directly filtering out the vast majority of low-probability, absurd words.

It acts as a safety fence: even if the temperature is set very high, the model cannot jump outside the candidate pool defined by Top-K to pick words randomly, effectively constraining the model from "going rogue."

3. Golden Parameter Pairing Schemes (Copy Directly for Your Business)

The two must be used in coordination; blindly maxing out or minimizing both is not an option:

1. ✅ Rigorous scenarios (code, contracts, knowledge base Q&A) Temperature = 0.1～0.3 + a moderately large Top-K. Low temperature ensures high-confidence words are prioritized; a larger Top-K retains a rich set of reasonable alternatives, balancing accuracy with expressive flexibility.
2. ✅ Creative scenarios (prose, scripts, copywriting, comic scripts) Temperature = 0.7～0.9 + a smaller Top-K. Moderately raising the temperature stimulates imagination; narrowing the Top-K range prevents the model from selecting words with extremely low probability and semantic disconnection, achieving "controlled creativity."

❌ Pitfall combination: High Temperature + High Top-K. The candidate pool is vast while the randomization force is maxed out, causing a sharp rise in the risk of hallucinations and incoherent speech. Avoid this in production environments.

4. Engineering Implementation: Building a Dual-Pipeline Workflow with LangChain

Why Choose LangChain?

LangChain = Language + Chain (LLM workflow orchestration) natively solves several development pain points:

1. PromptTemplate: Unified management and reuse of prompt templates, eliminating hardcoded prompts in code for easier iteration and maintenance;
2. OutputParser: Standardized parsing of the structure returned by the large model;
3. Chain (Pipeline): Connects the prompt template → LLM model → result parsing into a complete workflow;
4. Flexibly switch model instances, allowing the same business logic to easily differentiate between a "creative model" and a "rigorous model" pipeline.

Core idea: Instantiate two independent LLM configurations, one biased towards creative writing, the other towards rigorous realism, sharing a single prompt template and differentiating the output style solely through sampling parameters.

Complete Runnable Code (DeepSeek Model Example)

import 'dotenv/config';
import { ChatOpenAI } from '@langchain/openai'
import { StringOutputParser } from '@langchain/core/output_parsers';
import { PromptTemplate } from '@langchain/core/prompts'

/**
 * Creative model: for prose, copywriting, story creation
 * temperature 0.8 boosts randomness, stimulating creativity
 * topK=4 narrows the candidate word range, limiting the model from going too far off-track and controlling hallucinations
 */
const creativeModel = new ChatOpenAI({
    model: 'deepseek-v4-pro',
    temperature: 0.8,
    topK: 4,
    maxTokens: 600,
    apiKey: process.env.DEEPSEEK_API_KEY,
    configuration: {
        baseURL: 'https://api.deepseek.com',
    }
})

/**
 * Rigorous realistic model: knowledge output, formal documents, professional content
 * temperature 0.2 for conservative sampling, prioritizing high-probability standard expressions
 * topK=8 for a larger candidate pool, ensuring information completeness and rich expression
 */
const preciseModel = new ChatOpenAI({
    model: 'deepseek-v4-pro',
    temperature: 0.2,
    topK: 8,
    maxTokens: 600,
    apiKey: process.env.DEEPSEEK_API_KEY,
    configuration: {
        baseURL: 'https://api.deepseek.com',
    }
})

// Prompt template, centrally maintained and reusable in multiple places
const storyPrompt = PromptTemplate.fromTemplate(
`
Please write a short prose piece, theme: {theme}
Style: gentle and healing, around 200 words, no paragraph breaks, delicate text with a sense of imagery.
`
)

// Output parser, directly extracts text content, simplifying call result handling
const outputParser = new StringOutputParser();

// Assemble AI pipeline: Prompt template => LLM model => Result parsing
const creativeChain = storyPrompt
    .pipe(creativeModel)
    .pipe(outputParser)

const preciseChain = storyPrompt
    .pipe(preciseModel)
    .pipe(outputParser)

// Test execution
async function runDemo() {
    const theme = "Autumn mountain evening breeze";

    console.log("=====Creative Writing Mode=====");
    const creativeText = await creativeChain.invoke({ theme });
    console.log(creativeText);

    console.log("\n=====Rigorous Realistic Mode=====");
    const preciseText = await preciseChain.invoke({ theme });
    console.log(preciseText);
}

runDemo().catch(err => console.error("Execution exception:", err))

Expected Execution Results

• Creative pipeline: Nimble word choice, richer metaphors and imagery, noticeable differences in text with each run;
• Rigorous pipeline: Structured writing, objective and realistic descriptions, higher content similarity across multiple calls, rarely producing wildly imaginative expressions.

5. Production Best Practices Summary

1. Do not rely solely on raising Temperature to gain creativity; pair it with Top-K to define word selection boundaries and reduce hallucination risk;
2. Predefine multiple sets of LLM parameter configurations based on business scenarios, encapsulate them into different Chains, and switch as needed. Do not dynamically modify parameters arbitrarily at runtime;
3. For highly specialized businesses (customer service, knowledge bases, report generation), fix low-temperature parameters; for content creation businesses, enable the "high temperature + small TopK" combination;
4. Before going live, fully observe the output effects in a canary release. If semantic confusion occurs frequently, prioritize lowering the Temperature or narrowing Top-K;
5. Separate prompts from model configuration, using PromptTemplate for unified management to facilitate subsequent tuning and iteration.

Final Words

Many developers treat the instability of large model output as a black-box problem, but in reality, the randomness is entirely controlled by sampling parameters. Understanding the collaborative logic of Temperature and Top-K, and then using LangChain to build a standardized workflow pipeline, allows you to move from "passively accepting AI's random results" to actively controlling the AI's output style, adapting it to a wide variety of business scenarios.