How Temperature, Top-p, and Top-k Shape LLM Text Generation
See how temperature reshapes token probabilities while top-p and top-k narrow the candidates an LLM can choose from.
A Treasure Hunt
We begin by asking the language model:
Write one sentence about finding treasure.
Write one sentence about finding treasure.
The model reads the prompt, predicts one token, and adds it to the input. It then reads the updated input and predicts another token. This loop continues until it finishes the sentence:
Maya found a hidden chest.
In the generated sentence, the model chose the word found. But why did it choose found instead of discovered, uncovered, or opened?
To answer that, we need to look at the model’s vocabulary and understand how temperature, top-k, and top-p influence which token gets selected.
Meet Our Tiny LLM
Before an LLM can predict the next token, it needs a list of possible choices. That list is called its vocabulary.
A vocabulary is made of tokens. A token might be:
- A complete word, such as
dragon - Part of a word, such as
ing - Punctuation, such as
, - A number, symbol, or even a space pattern
Modern models such as GPT and Gemini can have vocabularies containing well over 100,000 tokens.
Every time one of these models generates the next piece of text, it produces a score for every token in that vocabulary. Watching more than 100,000 candidates compete would be impossible to visualize.
So, for our treasure hunt, we will use a tiny fictional LLM with only 30 tokens:
Maya, found, discovered, uncovered,
opened, followed, a, the, hidden,
ancient, golden, chest, map, key,
cave, trail, treasure, forest,
beneath, inside, near, and, was,
filled, with, gold, silver, ".",
",", dragon.
These 30 tokens are our model’s entire universe. Whenever it predicts the next token, it must score every one of them and select a winner.
How Is the Next Token Chosen?
Here is our two-part journey:
Part 1: Reshape the Competition
- 1 — Temperature: Change how strongly the model favors its most likely tokens without removing any candidates.
Part 2: Narrow the Field
- 2A — Top-p: Keep as many leading tokens as needed to reach a chosen probability.
- 2B — Top-k: Keep a fixed number of the most likely tokens.
For the rest of the article, we will focus on one step: how the model generated the second token, found, after Maya.
Maya found
We will follow found from its initial score to its final selection, watching how temperature, top-p, and top-k influence whether it gets chosen.
Let’s begin with temperature.
Part 1: Temperature Thaws the Competition
After reading Maya, our tiny LLM produces a score for every token in its vocabulary. These model-output scores are called logits.
| Token | Raw logit | Model preference |
|---|---|---|
| found | 6.75 | Favorite |
| discovered | 6.28 | Close contender |
| uncovered | 4.51 | Possible |
| opened | 3.74 | Less likely |
| dragon | −1.90 | Long shot |
A logit is not a probability. It only tells us how strongly the model favors one token over another.
To understand how temperature changes these choices, think of temperature as literally freezing or thawing the model’s strictness.
Under the hood, temperature adjusts each score:
You do not need to follow every calculation. The important part is what happens to the gaps between the candidates.
Turn It Down: The Competition Freezes
Below 1, dividing by a small temperature stretches the gaps.
For example, found originally leads discovered by:
At a temperature of 0.5, both logits are divided by 0.5:
The new gap is:
The original gap of 0.47 has doubled.
The competition becomes rigid. Found pulls away from the field and becomes increasingly difficult to beat.
Turn It Up: The Competition Thaws
Above 1, the gaps compress. The competition becomes more fluid.
At a temperature of 2, the same gap shrinks to 0.235.
Found remains the favorite, but it loses some of its advantage.
Lower-ranked tokens receive a larger chance to win the draw—even a long shot like dragon can occasionally slip through.
The ranking itself never changes. Temperature only changes how strongly that ranking controls the selection.
Feel the Temperature Change
Softmax converts the adjusted logits into probabilities that add up to 100%. Those probabilities determine each token’s chance of being selected.
The playground shows the 15 strongest candidates so the chart remains readable, but the probabilities are calculated across the model’s full 30-token vocabulary.
- Raw logit
- 6.75
- Adjusted logit
- 6.75
- Probability
- 54.48%
Move the slider and watch the gaps between the visible bars:
- Below 1: The gaps stretch. The competition freezes around found.
- At 1: Temperature leaves the logits unchanged, producing the baseline probability distribution.
- Above 1: The gaps compress. Found loses some of its advantage while lower-ranked candidates gain ground.
In our generation, found won the draw:
Maya found
Insight: Temperature controls the rigidity of the competition: cold locks onto the favorites; heat gives the rest of the field room to move.
Part 2A: Top-p Draws a Boundary
Temperature has finished its job. The model now has a probability for every token in its vocabulary.
But even the weakest tokens still have a small chance of being selected. Given enough generations, a long shot like dragon can eventually slip through.
Top-p trims away that unlikely tail while preserving the strongest alternatives. This reduces wildly implausible choices without forcing the model to select only its favorite.
Top-p looks at the probabilities, sorted from highest to lowest. Think of it as filling a probability bucket: tokens enter one by one until their combined probability reaches the chosen target, (p).
At temperature 1, our leading candidates look like this:
| Token | Probability | Cumulative probability |
|---|---|---|
| found | 54.48% | 54.48% |
| discovered | 34.05% | 88.53% |
| uncovered | 5.80% | 94.33% |
| opened | 2.69% | 97.01% |
Suppose we choose:
After found and discovered, the bucket contains 88.53%—not enough. Adding uncovered pushes it to 94.33%, crossing the target.
Top-p draws the boundary there:
found · discovered · uncovered
Everything after uncovered is excluded from this draw. The probabilities of the three remaining tokens are rescaled to add up to 100%, and the model samples from them.
Top-p: 0.90 · 3 tokens eligible · 94.33% retained
- Probability
- 54.48%
- Cumulative probability
- 54.48%
- Status
- Eligible
Move the slider and watch the boundary shift:
- Lower top-p: The bucket reaches its target with fewer tokens.
- Higher top-p: More tokens are needed to reach the target.
- Top-p of 1: The entire vocabulary remains eligible.
Top-p does not count tokens. It collects probability.
Top-p is also called nucleus sampling, a technique introduced in the 2019 paper The Curious Case of Neural Text Degeneration.
Think of a cell or comet: the nucleus is its dense core, while the tail is the thinner material stretching outward. Top-p keeps the dense, high-probability core of tokens and cuts away the less reliable tail.
Part 2B: Top-k Sets a Headcount Boundary
Top-p collects probability. Top-k counts tokens.
It sorts the vocabulary from most likely to least likely, then keeps the first (k) candidates. The cutoff does not care about their combined probability—only their rank.
| Token | Rank | Probability |
|---|---|---|
| found | 1 | 54.48% |
| discovered | 2 | 34.05% |
| uncovered | 3 | 5.80% |
| opened | 4 | 2.69% |
| followed | 5 | 0.88% |
Suppose we choose:
Top-k draws the boundary after the fourth token:
Kept: found · discovered · uncovered · opened
Removed: followed and everything below it
The four remaining probabilities are rescaled to add up to 100%, and the model samples from them.
Compare that with Top-p:
- With (p=0.90), Top-p kept three tokens because they already carried 94.33% of the probability.
- With (k=4), Top-k keeps four tokens because that is the requested headcount.
That is the distinction:
Top-p draws a flexible boundary based on probability.
Top-k draws a fixed boundary based on headcount.
Top-p adapts to the shape of the distribution. Top-k gives you a predictable maximum number of candidates.
Putting It All Together
The model begins with a probability distribution across its vocabulary. From there, generation has three jobs:
- Temperature reshapes the competition. It stretches or compresses the gaps between token probabilities without removing any candidates.
- Top-p or Top-k narrows the field.
- Top-p keeps enough tokens to reach a chosen amount of probability.
- Top-k keeps a fixed number of the highest-ranked tokens.
- Sampling chooses the winner from the tokens that remain.
In our generation, that winner was:
Maya found
Temperature, Top-p, and Top-k offer different kinds of control:
Temperature changes the odds.
Top-p draws a probability boundary.
Top-k draws a headcount boundary.
Concept Check
Vocabulary
The model’s universe of available tokens—the complete menu it can choose from.
Vocabulary. The model’s universe of available tokens—the complete menu it can choose from.
Quiz
86% of people love quizzes after learning. Are you one of them?
Question text
Quiz complete