Beginner-Friendly Topic
Take your time - it's perfectly normal to re-read this topic 2-3 times. Try the interactive code editor below to run code yourself. Use the Q&A section to check your understanding before moving on. You've got this! 🚀
Inference
Inference is the process of using a trained LLM to generate a response. Every time you send a message to ChatGPT or any AI tool, inference is happening.
What Happens During Inference
- Your prompt is tokenized (converted into tokens the model understands)
- The tokens are fed into the model along with any conversation history (all within the context window)
- The model processes all this through its learned layers
- For each generation step, the model computes probabilities for every possible next token
- A token is selected (typically the highest probability one, with some randomness)
- The selected token is added to the context and the process repeats
- Generation stops when the model produces a stop token or reaches the maximum output length
Temperature: Controlling Randomness
Most AI APIs have a 'temperature' setting that controls how random the output is.
Low temperature (0.0 to 0.3): always picks the highest-probability token. Output is more predictable and consistent. Good for factual answers and code.
High temperature (0.7 to 1.0): introduces more randomness, sometimes picking lower-probability tokens. Output is more varied and creative. Good for creative writing and brainstorming.
When you use ChatGPT, the temperature is set automatically depending on the task.
Deep Learning ⊂ Machine Learning ⊂ Artificial Intelligence
Key Takeaways
- Inference is the process of using a trained LLM to generate a response.
- Your prompt is tokenized (converted into tokens the model understands)
- The tokens are fed into the model along with any conversation history (all within the context window)
- The model processes all this through its learned layers