How LLMs Work: Simple Explanation
An LLM works by predicting the most likely next token given all the tokens that came before it. This one simple principle, applied at enormous scale, produces impressive language abilities.
The Core Idea: Next Token Prediction
At its core, an LLM does one thing: given a sequence of tokens, predict the most likely next token.
This sounds simple. But trained on billions of documents across every topic, the model builds a rich internal representation of language, knowledge, and reasoning.
The magic is that you do not just ask it to predict the next word in a sentence. You ask it questions, and the 'next word prediction' process is powerful enough to generate answers, solutions, and explanations.
This is similar to autocomplete on your phone, but trained on incomparably more data with a much more powerful model.
Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence
Simple Next-Token Prediction
# Illustrating next-token prediction (very simplified)
import random
random.seed(42)
# A tiny language model's learned knowledge
# In real LLMs, this is encoded in billions of parameters
# Here we use simple probability tables
learned_patterns = {
"The capital of France is": {"Paris": 0.97, "Lyon": 0.02, "Nice": 0.01},
"Paris is the capital of": {"France": 0.96, "Europe": 0.02, "Germany": 0.01, "Spain": 0.01},
"2 + 2 equals": {"4": 0.99, "5": 0.01},
"Python is a programming": {"language": 0.95, "snake": 0.04, "tool": 0.01},
"The largest planet is": {"Jupiter": 0.97, "Saturn": 0.02, "Neptune": 0.01},
}
def predict_next_token(context):
"""Predict the most likely next word given the context."""
if context in learned_patterns:
probs = learned_patterns[context]
# Sample from the probability distribution
words = list(probs.keys())
weights = list(probs.values())
return random.choices(words, weights=weights)[0]
return "[unknown]"
# Test the model
test_prompts = [
"The capital of France is",
"2 + 2 equals",
"Python is a programming",
"The largest planet is",
]
print("Next-Token Prediction (Simplified LLM):")
print()
for prompt in test_prompts:
next_token = predict_next_token(prompt)
print(f" Context: '{prompt}'")
print(f" Predicted next token: '{next_token}'")
print(f" Full prediction: '{prompt} {next_token}'")
print()Why This Simple Idea Works So Well
- Trained on billions of documents, the model sees language patterns from every domain
- Next-token prediction forces the model to understand context at a deep level to predict well
- The same model that predicts 'Paris' after 'The capital of France is' can predict the next word of code, poetry, or mathematical reasoning
- Scale matters enormously: larger models with more parameters make better predictions
- Fine-tuning with human feedback shapes the model to be helpful and safe
Key Takeaways
- An LLM works by predicting the most likely next token given all the tokens that came before it.
- Trained on billions of documents, the model sees language patterns from every domain
- Next-token prediction forces the model to understand context at a deep level to predict well
- The same model that predicts 'Paris' after 'The capital of France is' can predict the next word of code, poetry, or mathematical reasoning