The Core Challenge
AI systems can discriminate against people based on protected characteristics—not intentionally, but through biased training data, flawed metrics, or design that fails to account for different populations.
Key Concepts
| Algorithmic bias | Systematic errors in AI outputs that create unfair outcomes for particular groups. |
| Protected characteristics | Attributes where discrimination is illegal: race, gender, disability, age, religion, etc. |
| Proxy discrimination | When AI uses variables that correlate with protected characteristics (e.g., postcode as proxy for race). |
| Fairness metrics | Quantitative measures of whether a system treats different groups equitably. |
| Disparate impact | When a seemingly neutral system produces disproportionately adverse outcomes for protected groups. |
Warning Signs
Watch for these indicators of potential bias problems:
- Performance is only measured as overall accuracy, not across demographic groups
- Training data comes from historical decisions that may reflect past discrimination
- No bias testing was done before deployment
- The team building the system lacks diversity
- Stakeholders from affected communities weren't consulted
- Performance variation across groups is unknown or unmeasured
- There's no ongoing monitoring for bias drift after deployment
Questions to Ask in AI Project Reviews
- "What testing has been done for bias across different demographic groups?"
- "How representative is the training data? What populations might be underrepresented?"
- "What's our threshold for acceptable performance variation across groups?"
Questions to Ask in Governance Discussions
- "Who has authority to stop deployment if bias is found?"
- "What ongoing monitoring exists for bias drift over time?"
- "How are we meeting our obligations under the Public Sector Equality Duty?" (for public sector)
Questions to Ask in Strategy Sessions
- "What's our position on using AI in high-stakes decisions affecting individuals?"
- "How diverse are the teams designing and evaluating our AI systems?"
- "What transparency do we provide about known limitations and potential disparities?"
Reflection Prompts
- Your awareness: For AI systems in your area, do you know whether they perform differently for different groups?
- Your assumptions: What assumptions are you making about the fairness of AI systems you rely on?
- Your influence: What could you do to ensure bias is taken seriously in AI decisions you're involved in?
Good Practice Checklist
- Bias testing across demographic groups happens before deployment
- Multiple fairness metrics are tracked, not just overall accuracy
- Training data representativeness is documented and gaps acknowledged
- Diverse teams are involved in design and evaluation
- Ongoing monitoring detects bias drift after deployment
- Clear thresholds exist for acceptable performance variation
- There's transparency about known limitations with stakeholders
Quick Reference
| Element | Question to Ask | Red Flag |
|---|---|---|
| Testing | What bias testing was done? | "We checked overall accuracy" |
| Data | How representative is training data? | Unknown or assumed adequate |
| Metrics | What fairness metrics are tracked? | Only overall performance |
| Monitoring | How is bias drift detected? | No ongoing measurement |
| Authority | Who can stop deployment for bias? | No clear escalation path |
The Legal Context
Public Sector Equality Duty requires public organisations to actively eliminate discrimination. This applies to algorithmic systems.
Equality Act 2010 prohibits discrimination based on protected characteristics in employment, services, and public functions.
Emerging regulation will likely require impact assessments demonstrating fairness considerations for high-risk AI.