Skip to main content
Menu

glossary terms

Temperature

Category
LLMs & Generative AI
Difficulty
Intermediate

Definition

A hyperparameter that scales the logits before applying a softmax function, effectively controlling the probability distribution of the next token in a generative model.

How It Works and Context

In the context of Large Language Models (LLMs), temperature is a numerical value that dictates the 'creativity' or 'randomness' of the model's responses. When a model predicts the next token, it generates a list of probabilities for all possible candidates. The temperature parameter modifies these probabilities before the final selection is made. At a low temperature (near 0), the model becomes deterministic, consistently choosing the most likely token, which is ideal for factual tasks or coding. At a high temperature (above 1), the model flattens the probability distribution, making less likely tokens more probable, which encourages diverse, creative, or unpredictable output. However, setting the temperature too high can lead to 'hallucinations' or incoherent text, as the model may select highly improbable tokens that break the logical flow of the conversation.

Why It Matters

Temperature is a critical lever for developers and users to balance precision and creativity. In professional AI systems, choosing the right temperature is essential for task-specific performance: low settings ensure accuracy for data extraction and technical documentation, while higher settings are necessary for brainstorming, creative writing, and conversational variety. Understanding this parameter allows for fine-tuning the model's behavior to meet specific user needs without retraining the underlying architecture.

Real-world Example

Imagine a developer building a customer support bot. For the 'Technical Troubleshooting' module, they set the temperature to 0.1 to ensure the bot provides consistent, accurate, and safe instructions based on official documentation. Conversely, for the 'Marketing Copywriting' module, they set the temperature to 0.8, allowing the AI to generate more engaging, varied, and creative ad slogans that capture the brand's voice without sounding repetitive.

Common Mistakes

  • Assuming that a higher temperature always results in 'smarter' or more intelligent output.
  • Neglecting to adjust temperature when switching between factual and creative use cases.
  • Confusing temperature with 'Top-P' (nucleus sampling), which filters the candidate pool rather than scaling the probability distribution.
  • Setting the temperature to 0 in a creative task, leading to repetitive and robotic-sounding text.

Frequently Asked Questions

Does temperature affect the model's underlying knowledge?

No. Temperature does not change what the model knows; it only changes how the model selects from the probabilities it has already calculated based on its training data.

What happens if I set the temperature to exactly 0?

At a temperature of 0, the model becomes greedy, always selecting the single most likely next token. This makes the output deterministic, meaning the same prompt will consistently produce the exact same response.

Can temperature be used to fix hallucinations?

Lowering the temperature can reduce the likelihood of a model choosing 'creative' but incorrect tokens, which may help mitigate some types of hallucinations, but it cannot eliminate them entirely if the model lacks the correct information.

Related Glossary Terms

AI Hallucination

An AI hallucination occurs when a generative model produces output that is factually incorrect, nonsensical, or unfaithful to the provided source material while maintaining a tone of high confidence.

Generative AI

A class of artificial intelligence systems capable of generating new, original content—such as text, images, audio, or synthetic data—by learning the underlying patterns and structures of training datasets.

Inference

Inference is the process of using a trained machine learning model to make predictions or generate outputs based on new, unseen input data. It represents the operational phase of an AI system where the model applies the patterns learned during training to perform its intended task.

Large Language Model (LLM)

A type of artificial intelligence model trained on massive datasets using deep learning techniques, specifically the transformer architecture, to understand, summarize, and generate human-like text.

Tokenizer

A tokenizer is a computational component that breaks down raw text into smaller units called tokens, which are then mapped to numerical identifiers for processing by machine learning models.