Beginner-Friendly Topic
Take your time - it's perfectly normal to re-read this topic 2-3 times. Try the interactive code editor below to run code yourself. Use the Q&A section to check your understanding before moving on. You've got this! 🚀
Tokens
Tokens are the basic units that Generative AI models use to process text. Instead of reading character by character or word by word, AI models work with tokens, which are small chunks of text. Understanding tokens helps you understand model limits and costs.
What is a Token?
A token is a small piece of text that an AI model processes as a single unit.
Tokens are roughly:
- Short words: 'the', 'is', 'a' = 1 token each
- Longer words: 'hello' = 1 token, 'unhelpful' = 1 to 2 tokens
- Parts of long words: 'extraordinary' might be split into 'extra' + 'ordinary' = 2 tokens
- Punctuation: '.' ',' '!' = 1 token each
A rough guide: 100 tokens is approximately 75 English words, or roughly one short paragraph.
Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence
Why Tokens Matter
# Understanding tokens with a practical example
sentence = "The quick brown fox jumps over the lazy dog."
# Approximate tokenization (actual tokenization varies by model)
# In reality, models use algorithms like Byte-Pair Encoding (BPE)
approximate_tokens = sentence.split() # simplified: split by spaces
approximate_tokens.append(".") # add punctuation as separate token
print(f"Sentence: '{sentence}'")
print(f"Approximate token count: {len(approximate_tokens)}")
print(f"Tokens: {approximate_tokens}")
print()
# Tokens matter because:
print("Why tokens matter:")
print("1. Context window limit: models can only process a fixed number of tokens at once")
print("2. Cost: most AI API services charge by the number of tokens processed")
print("3. Speed: more tokens take more time to process")
print()
# Practical implications
context_window_example = 128000 # GPT-4's context window
tokens_per_page = 750 # approximate tokens per page of text
pages_in_window = context_window_example // tokens_per_page
print(f"A model with a {context_window_example:,} token context window")
print(f"can process approximately {pages_in_window} pages of text at once.")