Learning Rate
The learning rate controls how much the weights change after each batch. It is the single most important hyperparameter to tune in deep learning.
What the Learning Rate Does
After each training step, the network calculates how to adjust each weight. The learning rate determines how large that adjustment is.
Think of it as step size while hiking to a valley:
- Very large step: might jump over the valley and never reach the bottom
- Very small step: will eventually reach the bottom but takes forever
- Just right: reaches the valley efficiently
Typical starting values:
- 0.001 (1e-3) is a common default for the Adam optimizer
- 0.01 is common for SGD
- 0.0001 for fine-tuning pre-trained models
θ = θ - α × ∇L(θ). Too high α = diverge. Too low = slow.
Learning Rate Effects
# Illustrating the effect of learning rate on training
import random
random.seed(42)
def train_with_lr(learning_rate, epochs=30):
"""Simulate training with different learning rates."""
# Simple task: find weight = 2.0
weight = 0.0 # Start at 0
target = 2.0
losses = []
for _ in range(epochs):
gradient = -(target - weight) # simplified gradient
weight -= learning_rate * gradient
loss = (target - weight) ** 2
losses.append(loss)
return weight, losses[-1]
learning_rates = [0.001, 0.01, 0.1, 1.0, 2.5]
print("Effect of learning rate on training:")
print(f"{'Learning Rate':>15} {'Final Weight':>13} {'Final Loss':>12}")
print("-" * 50)
for lr in learning_rates:
final_weight, final_loss = train_with_lr(lr)
quality = "good" if final_loss < 0.01 else ("unstable" if final_loss > 10 else "slow")
print(f"{lr:>15.3f} {final_weight:>13.4f} {final_loss:>12.6f} ({quality})")
print()
print("Target weight: 2.0")
print("Too small LR: converges but slowly.")
print("Good LR: converges to target efficiently.")
print("Too large LR: oscillates and may diverge (unstable).")Tip
Tip
Start with a learning rate of 0.001 and see how training goes. If the loss does not decrease, try 0.01. If training is unstable (loss jumps around), try 0.0001. Most deep learning frameworks have learning rate schedulers that automatically reduce the rate as training progresses.