Training an LLM
Training an LLM requires vast amounts of data, enormous computing power, and a multi-stage process. Understanding this process helps you appreciate why LLMs work and what their limitations are.
The Three Stages of LLM Training
Stage 1: Pre-training
The model is trained on hundreds of billions of text tokens from the internet, books, code, and other sources. The task is simple: predict the next token. This stage gives the model its general knowledge and language abilities. It takes months and costs millions of dollars.
Stage 2: Supervised Fine-Tuning (SFT)
Human writers create examples of good (question, answer) pairs. The model is fine-tuned to produce helpful, accurate, and formatted responses.
Stage 3: Reinforcement Learning from Human Feedback (RLHF)
Human raters compare pairs of model outputs and choose which is better. A reward model is trained on these preferences. The LLM is then optimized to produce outputs that the reward model rates highly. This shapes the model to be helpful, harmless, and honest.
Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence
Training Data Scale
- GPT-3 was trained on approximately 570 GB of text (about 300 billion tokens)
- This includes Common Crawl (web pages), books, Wikipedia, and code repositories
- The model sees each token roughly once during training
- Training took thousands of GPU hours on hundreds of A100 GPUs
- GPT-4 and later models were trained on even larger, higher-quality datasets
- Data quality matters as much as quantity: modern models use careful filtering and deduplication
What Pre-training Teaches the Model
# Illustrating what a model learns from pre-training data
# A model trained on diverse text automatically learns:
learned_knowledge = {
"Language": [
"Grammar rules for many languages",
"Writing styles (formal, casual, technical)",
"Common phrases and idioms",
],
"World Knowledge": [
"History, science, culture, geography",
"Facts about countries, people, and events",
"Concepts from medicine, law, and finance",
],
"Reasoning": [
"How to follow logical steps",
"Mathematical relationships",
"Cause-and-effect relationships",
],
"Programming": [
"Syntax of many programming languages",
"Common algorithms and data structures",
"How to explain and debug code",
],
}
print("What an LLM Learns During Pre-training:")
print()
for category, examples in learned_knowledge.items():
print(f" {category}:")
for example in examples:
print(f" - {example}")
print()
print("All of this is learned from predicting the next token.")
print("The model never receives explicit lessons in any of these areas.")Key Takeaways
- Training an LLM requires vast amounts of data, enormous computing power, and a multi-stage process.
- GPT-3 was trained on approximately 570 GB of text (about 300 billion tokens)
- This includes Common Crawl (web pages), books, Wikipedia, and code repositories
- The model sees each token roughly once during training