AI Bias
AI bias occurs when an AI system produces systematically unfair results for certain groups of people. Understanding where bias comes from and how to detect it is one of the most important skills in responsible AI development.
Where Does Bias Come From?
AI bias almost always originates from one of three sources:
1. Biased training data
If the training data reflects historical discrimination, the model learns that discrimination. A hiring model trained on past decisions will mirror whoever was hired in the past—which may not reflect who should be hired.
2. Biased labels
If humans labeled the training data with biased judgments, those biases are embedded into the model. Human labeling is never perfectly objective.
3. Biased feature selection
Some features may seem neutral but correlate with protected characteristics. Zip code may seem neutral but can act as a proxy for race. University attended may correlate with socioeconomic background.
Technical diagram.
Real-World AI Bias Examples
- COMPAS criminal risk tool: found to be twice as likely to incorrectly flag Black defendants as high-risk compared to white defendants
- Amazon resume screening (2018): trained on historically male-dominated hiring data, the model penalised CVs mentioning women's organisations
- Facial recognition: multiple commercial systems had significantly higher error rates for darker-skinned women than lighter-skinned men
- Healthcare algorithm: a widely used hospital algorithm prioritised white patients over sicker Black patients because it used historical spending as a proxy for medical need
- Social media recommendation: algorithmic amplification of outrage-generating content disproportionately impacts vulnerable communities
Detecting Bias in Code
# Detecting demographic bias in model outputs
def check_for_bias(predictions, groups, group_a="Group A", group_b="Group B"):
"""
Compare approval rates between two groups.
Simple check for disparate impact.
"""
a_preds = [p for p, g in zip(predictions, groups) if g == group_a]
b_preds = [p for p, g in zip(predictions, groups) if g == group_b]
a_rate = sum(a_preds) / len(a_preds) if a_preds else 0
b_rate = sum(b_preds) / len(b_preds) if b_preds else 0
print(f"Approval Rates:")
print(f" {group_a}: {a_rate*100:.1f}%")
print(f" {group_b}: {b_rate*100:.1f}%")
print(f" Gap: {abs(a_rate - b_rate)*100:.1f} percentage points")
print()
# 80% rule (4/5 rule): if minority rate < 80% of majority rate, potential bias
if a_rate > 0:
ratio = b_rate / a_rate
print(f" Disparate Impact Ratio: {ratio:.2f}")
if ratio < 0.80:
print(f" WARNING: Potential bias detected (ratio < 0.80)")
print(f" {group_b} is approved at less than 80% of the rate of {group_a}")
else:
print(f" OK: No significant disparate impact detected")
# Example: loan approval predictions
import random
random.seed(42)
groups = ["Group A"] * 100 + ["Group B"] * 100
# Biased model: Group B is approved less often even when equally qualified
predictions = []
for g in groups:
if g == "Group A":
predictions.append(1 if random.random() > 0.35 else 0)
else:
predictions.append(1 if random.random() > 0.55 else 0) # bias here
check_for_bias(predictions, groups)Common Mistake
Warning
Removing the protected attribute (race, gender, age) from the training data does NOT prevent bias. If other features correlate with the protected attribute—for example, zip code, school attended, or name—the model will still learn to discriminate indirectly. This is called proxy discrimination and it is extremely common.
Key Takeaways
- AI bias occurs when an AI system produces systematically unfair results for certain groups of people.
- COMPAS criminal risk tool: found to be twice as likely to incorrectly flag Black defendants as high-risk compared to white defendants
- Amazon resume screening (2018): trained on historically male-dominated hiring data, the model penalised CVs mentioning women's organisations
- Facial recognition: multiple commercial systems had significantly higher error rates for darker-skinned women than lighter-skinned men