Tokens
Tokens are the basic units that LLMs work with. Understanding what a token is helps you understand how LLMs process and generate text, and why token count matters for cost and context limits.
What is a Token?
A token is a piece of text that the LLM processes as a single unit. Tokens are not exactly words.
In most LLMs:
- Common short words are one token: 'the', 'cat', 'is'
- Longer words are split into multiple tokens: 'fantastic' might be 'fan' + 'tastic'
- Punctuation is often its own token: '!', '.', ','
- Numbers are split digit by digit: '123' might be '1' + '2' + '3'
- Non-English text often uses more tokens per word than English
A rule of thumb: 1 token is roughly 0.75 English words, or about 4 characters.
Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence
Why Token Count Matters
- Cost: API providers charge per token (both input and output tokens)
- Context window: each LLM can process a maximum number of tokens at once
- Speed: more tokens = slower generation and higher memory usage
- Context length affects what the model can 'remember' in a conversation
- Rule of thumb: 1000 tokens is roughly 750 words or about 3 pages of text
Token Counting Example
# Estimating token counts for different texts
def estimate_tokens(text):
"""
Rough token estimation.
Real count depends on the specific tokenizer.
Rule of thumb: 1 token per 4 characters (for English).
"""
char_count = len(text)
word_count = len(text.split())
# Two common heuristics:
by_chars = char_count / 4
by_words = word_count / 0.75
return round((by_chars + by_words) / 2)
# Estimate tokens for different text types
texts = [
("Short prompt", "What is machine learning?"),
("Medium prompt", "Explain the concept of overfitting in machine learning. What causes it and how can it be prevented? Include practical examples."),
("Long prompt", """
Write a comprehensive 500-word article explaining Artificial Intelligence
for a complete beginner. Include what AI is, examples from everyday life,
the difference between AI and machine learning, and why AI matters for
the future of technology and society.
""".strip()),
("Simple code", "def hello(): print('Hello World')"),
("GPT-3 full training text (estimate)", "All text on the internet up to early 2020"),
]
print("Token Count Estimates:")
print()
for name, text in texts[:-1]:
tokens = estimate_tokens(text)
chars = len(text)
print(f" {name}:")
print(f" Characters: {chars}")
print(f" Estimated tokens: {tokens}")
print()
print("Context window comparison:")
models = {
"GPT-3.5": "16,000 tokens",
"GPT-4o": "128,000 tokens",
"Claude 3.5": "200,000 tokens",
"Gemini 1.5 Pro": "1,000,000 tokens",
}
for model, context in models.items():
print(f" {model}: {context}")Tip
Tip
When using an LLM API, always consider the token count of both your input (prompt) and the expected output (generation). OpenAI's tokenizer page lets you paste text and see the exact token count. This is useful for estimating costs before sending large amounts of text.
Key Takeaways
- Tokens are the basic units that LLMs work with.
- Cost: API providers charge per token (both input and output tokens)
- Context window: each LLM can process a maximum number of tokens at once
- Speed: more tokens = slower generation and higher memory usage