Beginner-Friendly Topic
Take your time - it's perfectly normal to re-read this topic 2-3 times. Try the interactive code editor below to run code yourself. Use the Q&A section to check your understanding before moving on. You've got this! 🚀
Activation Functions
Activation functions decide whether a neuron should fire (be active) based on its input. They also add non-linearity, which is what makes neural networks able to learn complex patterns.
Why Activation Functions Are Needed
Without activation functions, no matter how many layers you add, the entire network just calculates a linear combination of inputs. This means it can only learn straight-line relationships.
Activation functions introduce non-linearity, allowing networks to learn curved, complex decision boundaries. This is what gives neural networks the power to recognize faces, translate languages, and generate images.
Input → Hidden layers → Output. Train via backpropagation.
Common Activation Functions
- ReLU (Rectified Linear Unit): output = max(0, input). Simple and widely used in hidden layers
- Sigmoid: squashes output to 0-1. Used in binary classification output layers
- Softmax: converts outputs to probabilities that sum to 1. Used in multi-class classification output
- Tanh: squashes output to -1 to 1. Sometimes used in hidden layers
- ReLU is the default choice for hidden layers in most networks today
Activation Functions in Code
import math
# The three most common activation functions
def relu(z):
"""ReLU: output is z if positive, otherwise 0."""
return max(0, z)
def sigmoid(z):
"""Sigmoid: squashes any number to (0, 1)."""
return 1 / (1 + math.exp(-z))
def softmax(z_list):
"""Softmax: converts list of numbers to probabilities summing to 1."""
exps = [math.exp(z) for z in z_list]
total = sum(exps)
return [e / total for e in exps]
# Test values
test_values = [-3, -1, 0, 1, 3, 5]
print("z value | ReLU | Sigmoid")
print("-" * 35)
for z in test_values:
print(f" {z:5} | {relu(z):4.2f} | {sigmoid(z):.4f}")
print()
# Softmax example (for multi-class classification)
raw_outputs = [2.0, 1.0, 0.1] # 3 classes
probs = softmax(raw_outputs)
classes = ["Cat", "Dog", "Bird"]
print("Softmax output (probabilities sum to 1):")
for name, prob in zip(classes, probs):
print(f" {name}: {prob:.3f} ({prob*100:.1f}%)")
print(f" Total: {sum(probs):.3f}")Try It Yourself
Key Takeaways
- Activation functions decide whether a neuron should fire (be active) based on its input.
- ReLU (Rectified Linear Unit): output = max(0, input). Simple and widely used in hidden layers
- Sigmoid: squashes output to 0-1. Used in binary classification output layers
- Softmax: converts outputs to probabilities that sum to 1. Used in multi-class classification output