Predicting the Next Token
Every word in every LLM response is generated one token at a time, with the model computing probabilities for the entire vocabulary before selecting the next token.
How One Token Is Generated
When you send a message to an LLM, here is what happens:
- 1Your message is tokenized: split into tokens
- 2Each token is converted to an embedding vector
- 3All tokens pass through many Transformer layers
- 4The final layer produces a score for every token in the vocabulary
- 5Scores are converted to probabilities using Softmax
- 6One token is selected (via sampling or greedy decoding)
- 7That token is added to the sequence
- 8Steps 2 to 7 repeat until the response is complete
For a 200-token response, this loop runs 200 times.
Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence
Temperature: Controlling Randomness
Temperature is a setting that controls how 'creative' vs 'predictable' the model is.
Temperature = 0: always picks the most likely token. Deterministic, repetitive
Temperature = 1.0: samples proportionally from probabilities. Natural and varied
Temperature > 1.5: amplifies unlikely choices. More creative but can become incoherent
For factual tasks (math, code, data extraction): use low temperature (0 to 0.2)
For creative tasks (stories, brainstorming, dialogue): use higher temperature (0.7 to 1.0)
Temperature in Code
import random
import math
def softmax_with_temperature(logits, temperature=1.0):
"""Apply temperature to logits and convert to probabilities."""
if temperature == 0:
# Greedy: pick the top token
result = [0.0] * len(logits)
result[logits.index(max(logits))] = 1.0
return result
scaled = [l / temperature for l in logits]
max_val = max(scaled)
exps = [math.exp(l - max_val) for l in scaled]
total = sum(exps)
return [e / total for e in exps]
# Simulate token probabilities from model output
# Higher logit = more likely
vocabulary = ["cat", "dog", "bird", "fish", "hamster"]
raw_logits = [5.0, 3.0, 1.5, 0.5, 0.3] # cat is most likely
print("Effect of Temperature on Token Probabilities:")
print()
print(f"Tokens: {vocabulary}")
print(f"Raw logits: {raw_logits}")
print()
temperatures = [0.1, 0.5, 1.0, 1.5]
for temp in temperatures:
probs = softmax_with_temperature(raw_logits, temp)
formatted = [f"{t}: {p:.3f}" for t, p in zip(vocabulary, probs)]
winner = vocabulary[probs.index(max(probs))]
print(f"Temperature {temp}: {', '.join(formatted)}")
print(f" Most likely: '{winner}' (prob: {max(probs):.3f})")
print()
print("Low temperature: 'cat' dominates. High temperature: others get more chance.")