How Computers Understand Text
Computers process numbers, not words. Before any NLP model can work with text, the text must be converted into numbers. This is the fundamental challenge of NLP.
The Core Challenge
Humans read words. Computers process numbers. The entire field of NLP is built around solving this mismatch.
The basic pipeline for text processing:
1. Get raw text: 'The movie was fantastic!'
2. Tokenize: split into tokens ['The', 'movie', 'was', 'fantastic', '!']
3. Encode: convert each token to a number [1, 234, 45, 892, 5]
4. Embed: convert each number to a rich numerical vector [... 512 numbers per token ...]
5. Process: pass the vectors through neural network layers
6. Output: produce a label, a generated text, or other result
Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence
Text to Numbers
# Demonstrating text to number conversion
# Vocabulary: a mapping of words to unique numbers
vocabulary = {
"<PAD>": 0, # padding token (for batching)
"<UNK>": 1, # unknown word token
"the": 2,
"movie": 3,
"was": 4,
"fantastic": 5,
"terrible": 6,
"good": 7,
"bad": 8,
"I": 9,
"loved": 10,
"hated": 11,
"it": 12,
}
def text_to_numbers(text, vocab):
"""Convert text to a sequence of numbers."""
words = text.lower().split()
numbers = [vocab.get(word, vocab["<UNK>"]) for word in words]
return numbers
# Convert sentences
sentences = [
"the movie was fantastic",
"I loved it",
"the movie was terrible",
"I hated it",
]
print("Text to Numbers Conversion:")
print()
for sentence in sentences:
numbers = text_to_numbers(sentence, vocabulary)
print(f" '{sentence}'")
print(f" -> {numbers}")
print()
print("The model processes these number sequences, not the raw text.")
print("Words with similar meanings should ideally get similar number vectors.")
print("This is what word embeddings achieve (more on this in Module 9 on LLMs).")Common Mistake
Warning
A vocabulary only covers words it was trained on. New words, slang, and technical terms not in the vocabulary are assigned an 'unknown' token and the model cannot understand them well. Modern tokenizers like BPE (Byte-Pair Encoding) handle this by breaking words into smaller sub-word pieces.