Should I set both temperature and top-p?

Usually pick one and leave the other at its default. They shape the same list of odds in different ways, and moving both makes it hard to reason about what a change did. Some providers also document that they should not be changed together.

Is temperature 0 deterministic?

Nearly, and both major vendors say exactly that in their own documentation. At zero the top token is picked every time, but batching and hardware differences under load can still change which token is top by a hair. A test that asserts byte identical output will eventually fail for no reason you can find.

Does a high temperature make the model more creative?

It makes the draw more willing to pick tokens the model itself rated as unlikely. A little of that reads as range; past a point it reads as noise, because the model was right that those tokens were bad fits.

Is 0.7 the same setting everywhere?

No. Providers publish different ranges, so the same number can sit at different points on the scale. Read the range for the API you are calling before you copy a value from a blog post, including this one.

ExplainerAI Fundamentals

Temperature vs top-p, watch the sampling change

The model hands over odds; code after it draws a token. Watch temperature reshape those odds without reordering them, and top-p cut the tail out of the draw.

4 Sept 20263 min read

Temperature vs top-p, watch the sampling change, the opening step of the interactive scene — The model hands over odds, not a word
For one prompt it scores every token in its vocabulary. The scores become shares that add up to one whole. Then it stops. It never chooses.
Step 1 of 6

Ask a model the same question twice and get two different answers, and nothing about the model changed. It did not change its mind, because it never made one. At every step it hands over odds for every token it knows, and a piece of ordinary code after it draws one. Step through the scene above to watch the two knobs that reshape those odds.

The model forecasts, something else decides

For one prompt, the model scores every token in its vocabulary. The scores become shares that add to one, and that list is the model's entire output at that step. The draw happens afterwards, in code you control:

// the shape of every sampler, stripped down
const shares = softmax(logits.map((score) => score / temperature));
const nucleus = smallestSetReaching(shares, topP);   // top-p keeps the head
const token = drawOne(nucleus);                       // a weighted random pick

If cat holds two thirds of the share, it lands about two draws in three. A percentage describes many draws, not this one, which is why one prompt can give you cat now and dog a minute later with the model untouched.

Two knobs, two different edits to the same list

Temperature divides the scores before they become shares. Below one, the gaps widen: the favourite takes more of the pie, the tail shrinks toward nothing, and at zero the top token takes all of it, which is greedy decoding and no longer a draw at all. Above one, the same rows flatten toward each other and the tail gets picked more often. In both directions every share changes size and not one changes place. The favourite stays the favourite.

Top-p decides who is allowed in the draw. Sort the tokens by share, keep the smallest set whose shares add up to p, and draw from those only. At 0.9, a confident step might keep one token and an open ended one might keep a dozen, because the cut is a share of the probability, not a fixed count. The tokens outside keep their odds and their order; they are simply not eligible. That is nucleus sampling, from the paper that introduced it, and it exists because the long tail is where the strange words come from.

Two things worth knowing on sight. Temperature 0 is nearly repeatable, not guaranteed: both major vendors say so in their own docs, so build tests on "nearly". And repeatable is not the same as correct; a wrong answer at temperature zero is wrong every time. The lesson runs every draw from a recorded forecast of a real model, so the odds you drag around there are its actual numbers.

Common questions

Should I set both temperature and top-p?: Usually pick one and leave the other at its default. They shape the same list of odds in different ways, and moving both makes it hard to reason about what a change did. Some providers also document that they should not be changed together.
Is temperature 0 deterministic?: Nearly, and both major vendors say exactly that in their own documentation. At zero the top token is picked every time, but batching and hardware differences under load can still change which token is top by a hair. A test that asserts byte identical output will eventually fail for no reason you can find.
Does a high temperature make the model more creative?: It makes the draw more willing to pick tokens the model itself rated as unlikely. A little of that reads as range; past a point it reads as noise, because the model was right that those tokens were bad fits.
Is 0.7 the same setting everywhere?: No. Providers publish different ranges, so the same number can sit at different points on the scale. Read the range for the API you are calling before you copy a value from a blog post, including this one.

ExplainerAI Fundamentals

What is a context window, and why did it forget my name

A model has no memory between requests. Watch your app resend the whole chat every turn, fill the window, and drop the very turn that held your name.

4 Sept 20263 min read

ExplainerBackend

Blocking vs non blocking in Node, watch the thread pool

One readFileSync freezes every user on your server. Watch the thread stop on one line, then hand the read to a worker and answer everyone else meanwhile.

4 Sept 20263 min read