What Temperature Does in a Language Model

To put it simply, temperature is a dial that decides how brave a language model is allowed to be when it picks its next word. Turn it down and the model plays safe, choosing the word it believes in most, almost every time. Turn it up and it starts taking chances on words it only half believes in. Same model, same training, same knowledge. Only the bravery changes.

A small scene from my own desk. I once asked a model to finish the line "The cat sat on the" ten times, with the setting turned low, and it answered "mat" ten times in a row. Then I turned the setting up and asked again. This time I got "mat", "floor", "windowsill", and once, memorably, "keyboard". Nothing inside the model had changed between those two runs. It knew the same words, and it believed in them in exactly the same order. The only thing I had moved was how far it was allowed to stray from its first choice.

The whole article on one card: one dial, sitting between the model's scores and the word it finally picks.

The model scores every possible next word

A language model writes one word at a time — strictly, one token at a time, where a token is a word or a piece of a word. Before it writes each one, it looks at everything so far and does something most people never see: it gives a score to every word it knows. These raw scores have a name, logits, and you can think of them as the model's gut feeling, written down as numbers. "Mat" gets a high score. "Floor" gets a lower one. "Banana" gets a score so low it is almost a joke.

Scores are awkward to work with, so a function called softmax turns them into chances — clean percentages that add up to exactly 100. After "The cat sat on the", the chances might look like this: "mat" 63 percent, "floor" 23 percent, "sofa" about 9 percent, "roof" 5 percent, and a long tail of stranger words sharing the crumbs. That list is the model's honest opinion about what comes next.

Then comes the step this whole article is about: picking. The model does not have to take the top word. In normal use it samples — it picks at random, but not fairly. Each word's chance of being picked is exactly its percentage. "Mat", at 63 percent, gets picked most of the time. "Roof", at 5, sneaks in once in a while, and the sentence wanders off somewhere you did not expect. Temperature is the setting that reshapes those percentages just before the pick.

Temperature stretches or squeezes the chances

Here is the whole mechanism, and it is smaller than people expect. Before softmax runs, every score is divided by the temperature. That is it. One division. But division changes the shape of the result in a way worth understanding.

Dividing by a small number, like 0.5, makes every score bigger — and, more importantly, makes the gaps between scores wider. The favourite pulls further ahead of the pack. After softmax, its chance grows and everyone else's shrinks. Dividing by a big number, like 2, squeezes the scores together. The gaps narrow, the underdogs catch up, and words the model only half believed in suddenly get a fair hearing. A temperature of exactly 1 leaves the scores untouched: the model's raw opinion, unedited.

It works a lot like ordering noodles and choosing your chilli level. Mild, and the bowl tastes the way it always tastes — safe, familiar, a little plain. Extra hot, and the same bowl becomes an adventure: exciting, surprising, and once in a while a genuine mistake. The cook did not change. The noodles did not change. Only the level you asked for changed.

Here is the flow in a picture, because this is the part worth fixing in your head:

Four steps happen before every single word a model writes. Temperature lives in step two.

And here is the same idea as a few lines of Python, so you can watch the chances move yourself:

import numpy as np

def softmax(logits, temperature):
    scaled = np.array(logits) / temperature  # temperature lives in this one line
    scaled = scaled - scaled.max()           # keep the numbers small and safe
    exp = np.exp(scaled)
    return exp / exp.sum()

scores = [3.0, 2.0, 1.0, 0.5]  # mat, floor, sofa, roof

for t in (0.5, 1.0, 2.0):
    print(t, np.round(softmax(scores, t), 3))

Run it and three lines come back. At temperature 0.5, "mat" climbs from 63 percent to about 86, and "roof" nearly vanishes at less than 1 percent. At 1.0, nothing changes — 63, 23, 9, 5, the model's own opinion. At 2.0, "mat" falls to 44 percent while "roof" rises to 13. Same scores in, three very different sets of chances out. The subtraction in the middle line is only housekeeping — it stops the exponentials from growing into numbers too big for the computer to hold — but the division in the first line is the entire dial.

Temperature zero means always take the favourite

Push the idea to its limit and you get temperature 0. Dividing by a number that tiny would stretch the gaps so far that the favourite takes everything — so in practice, systems treat 0 as a simple rule: forget the random pick, always take the highest-scoring word. This is called greedy decoding.

Greedy has real virtues. Ask the same question twice and you get the same answer twice, which is exactly what you want when a program is pulling a date out of an invoice or filling in a form. But greedy text also goes flat fast. A model that always takes its safest word starts repeating itself — the same phrase, the same sentence shape, sometimes the same sentence, in a loop. Safe is not the same as good. It is just safe.

Top-k and top-p trim the menu instead

Temperature has two companion settings, and they solve a nearby problem: sometimes you do not want to reshape the chances, you want to remove the silly options entirely. Top-k keeps only the k best-scoring words — say the top 40 — and throws the rest away before the pick. Top-p, also called nucleus sampling, keeps the smallest group of words whose chances add up to some share, like 90 percent, and cuts the rest. On a confident turn, that group might be three words. On an open turn, it might be three hundred.

In practice the three settings work as a team. Top-k and top-p decide who is allowed into the room. Temperature decides how boldly the model chooses among the ones inside.

Which setting for which job

So what should you actually use? The honest answer is that it depends on what a wrong word costs you. Some rough settings that work well in daily use:

• Code, maths, and anything you will run or submit: 0 to 0.3. You want the model's first answer, the same way every time.
• Facts, summaries, and pulling details out of a document: 0.2 to 0.5. A little freedom, no adventures.
• Everyday chat, emails, and drafts: 0.6 to 0.8. Natural variety without wandering off.
• Stories, poems, names, and brainstorming: 0.9 to 1.2. This is where the surprises you actually want live.
• Above 1.3 or so: expect drift, invented words, and broken grammar. Fun to play with, risky to ship.

One more thing worth knowing: the dial is applied again at every single word. A short answer barely feels it. A long answer feels it hundreds of times, and small chances compound — which is why a warm setting can start a story sensibly and be somewhere quite strange by page two.

One warning matters more than the numbers. Temperature cannot fix a model that does not know something. Turning the heat up does not create knowledge — it just makes the guessing bolder. A wrong fact at low temperature is a wrong fact stated plainly. The same wrong fact at high temperature is a wrong fact wearing a costume. If the answer matters, the fix is better information in the prompt and checking afterwards, not a luckier dial.

Here is the whole dial in one picture — the same four words from the code above, at two settings of the range:

Same model, same sentence, same scores. The only thing that moved is the temperature.

Takeaways

• A model scores every possible next word, softmax turns the scores into chances, and sampling picks one — temperature reshapes the chances in between.
• Low temperature widens the gaps: the favourite wins almost always, so text is steady, factual, and sometimes repetitive.
• High temperature levels the field: unlikely words get their turn, so text is varied, creative, and riskier.
• Temperature 0 is greedy decoding — always take the top word, no randomness, no surprises.
• Top-k and top-p trim which words may be picked at all; temperature decides how boldly the model picks among the survivors.
• Match the dial to the cost of a wrong word: cold for code and facts, warm for stories and ideas.

References and further reading

• Softmax function — Wikipedia
• Boltzmann distribution — Wikipedia
• Generation — Hugging Face Transformers documentation
• Distilling the Knowledge in a Neural Network — Hinton, Vinyals and Dean, arXiv:1503.02531

— Min SiThu Software Engineer & Tech Content Creator https://minsithu.org