AI Bias Testing: How to Comply with EU AI Act Article 10

Article 10 of the EU AI Act requires providers of high-risk AI systems to examine their training, validation, and testing datasets for possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights, or lead to discrimination prohibited under EU law — and to take appropriate measures to detect, prevent, and mitigate those biases. The regulation is explicit about the obligation. It is silent about the method. It tells you what you must do but not how to do it. This guide fills that gap. It walks you through the bias types you need to test for, the fairness metrics available, the open-source tools that implement them, and the documentation you need to produce for conformity assessment. If you are building, training, fine-tuning, or deploying a high-risk AI system, bias testing is not optional — it is a legal prerequisite.

• Legal basis: Article 10(2)(f) requires you to examine datasets for possible biases and take appropriate measures to detect, prevent, and mitigate them.
• Protected data exception: Article 10(5) permits processing special category data (race, gender, disability, etc.) strictly for the purpose of bias monitoring, detection, and correction — subject to safeguards.
• Six bias types to test for: historical, representation, measurement, aggregation, evaluation, and deployment bias.
• Protected attributes derive from the EU Charter of Fundamental Rights and GDPR Article 9: racial/ethnic origin, gender, age, disability, religion, sexual orientation, political opinion, and more.
• Fairness metrics are not interchangeable. Demographic parity, equalized odds, equal opportunity, predictive parity, and calibration each measure different aspects — and it is mathematically impossible to satisfy all simultaneously.
• Tooling exists: Fairlearn, AI Fairness 360, Aequitas, the What-If Tool, and Responsible AI Toolbox are mature, open-source options.
• Documentation is mandatory: Annex IV Section 2 requires you to document data examination measures, bias detection methodology, and remediation steps.
• Production monitoring: Bias testing is not a one-time gate — it must continue throughout the system's lifecycle under Article 9 risk management requirements.

What Article 10 actually requires

Article 10 establishes a comprehensive data governance regime for high-risk AI systems. Within that regime, the bias-specific obligations are concentrated in three provisions:

Article 10(2)(f) — Examination for biases: Training, validation, and testing datasets must be subject to examination in view of possible biases that are likely to affect the health and safety of persons, have a negative impact on fundamental rights, or lead to discrimination prohibited under Union law. This is not a suggestion. The word used is "shall." The examination must cover the datasets themselves — not just the model outputs.

Article 10(2)(f) continued — Appropriate measures: Following examination, the provider must take appropriate measures to detect, prevent, and mitigate those biases. "Appropriate" means proportionate to the risk level, the state of the art, and the specific context of the AI system. What counts as appropriate for a spam filter is different from what counts as appropriate for a credit scoring system or an HR recruitment tool.

Article 10(5) — Special category data for bias correction: This is the provision most teams overlook. To the extent strictly necessary for bias monitoring, detection, and correction, providers of high-risk AI systems may process special categories of personal data — the data types normally prohibited under GDPR Article 9: racial origin, ethnic origin, political opinions, religious beliefs, trade union membership, genetic data, biometric data, health data, sex life, and sexual orientation. This processing is permitted only subject to appropriate safeguards for the fundamental rights and freedoms of natural persons, including technical limitations on re-use and use of state-of-the-art security and privacy-preserving measures such as pseudonymisation or encryption where anonymisation may significantly affect the purpose pursued.

This is significant. The AI Act carves out a legal basis for processing otherwise prohibited data when the purpose is fairness monitoring. But the safeguards are strict: data must be pseudonymised or encrypted, access restricted, re-use forbidden, and processing strictly necessary — not merely useful.

The connection to other requirements is tight. Article 9 requires you to identify bias risks in your risk management system. Article 11 requires documentation of your bias testing methodology. Article 15 requires performance metrics that implicitly include fairness metrics. And the FRIA obligation for deployers specifically targets fundamental rights impacts — many of which trace back to biased systems.

Six types of AI bias you must test for

Bias in AI is not a single phenomenon. It enters the pipeline at different stages, from different sources, and manifests in different ways. Testing for "bias" generically is insufficient — you need to identify which types are relevant to your system and apply the appropriate detection methods for each.

Historical bias

Definition: Historical bias occurs when the data faithfully reflects the real world, but the real world itself contains systemic inequality. The data is accurate — the problem is that reality is biased, and the model learns to replicate that bias.

Detection requires domain expertise and external reference data — comparing model outcomes against a known fair baseline rather than against the training distribution.

Representation bias

Definition: Representation bias occurs when certain groups are underrepresented or overrepresented in the dataset relative to the population the system will serve. The model learns a detailed picture of majority groups and a blurry, unreliable picture of minority groups.

Detection requires demographic composition analysis: compare group proportions in the data against the target deployment population and measure performance metrics disaggregated by demographic group.

Measurement bias

Definition: Measurement bias occurs when the features or labels used to train the model are measured differently across groups, or when proxy variables correlate with protected attributes in ways that create systematic distortion.

Aggregation bias

Definition: Aggregation bias occurs when a single model is applied to groups that have fundamentally different characteristics, and the aggregation masks important within-group patterns. The model optimises for the average, which may not represent any specific group well.

Evaluation bias

Definition: Evaluation bias occurs when the benchmark dataset or evaluation methodology does not represent the real-world deployment population, causing the model to appear fairer or more accurate than it actually is in practice.

Deployment bias

Definition: Deployment bias occurs when a system is used in a context or manner different from what it was designed and tested for, creating biased outcomes that were not present during development.

Detection requires intended-use documentation and human oversight mechanisms that flag when the system is used outside its validated parameters.

Protected attributes under EU law

The EU AI Act does not define its own list of protected attributes. Instead, it references the EU Charter of Fundamental Rights (particularly Article 21 on non-discrimination) and aligns with GDPR Article 9 special categories. Together, these instruments establish the attributes you must test for:

Proxy variables matter as much as direct attributes. A model that does not ingest "gender" as a feature can still discriminate based on gender if it uses proxy variables — such as "job title" (correlated with gender due to occupational segregation) or "first name" (highly predictive of gender). Your bias testing must account for both direct and indirect discrimination, consistent with the FRIA methodology which requires assessing indirect fundamental rights impacts.

Fairness metrics explained

There is no single definition of "fair." Different fairness metrics formalise different ethical intuitions, and — crucially — it has been mathematically proven that most fairness definitions cannot be satisfied simultaneously except in trivial cases. You must choose which metrics are most appropriate for your use case.

How to choose

The choice of metric depends on the domain, the consequences of errors, and the rights at stake:

• Hiring and recruitment (HR compliance guide): Start with demographic parity for shortlisting stages (equal access to opportunity) and equal opportunity for final screening (don't miss qualified candidates from any group).
• Credit scoring and lending: Use equalized odds (both false approvals and false rejections carry harm) combined with calibration (scores must mean the same thing regardless of group).
• Healthcare diagnostics: Prioritise equal opportunity (every patient with a condition should be detected) and calibration (a 70% probability of disease should be 70% for all demographic groups).
• Criminal justice and law enforcement: Use equalized odds with particular scrutiny of false positive rates across racial and ethnic groups — false accusations carry severe fundamental rights consequences.
• Insurance and benefits: Use predictive parity (if the system flags someone as high-risk, the probability of actual risk should be equal across groups) combined with calibration.

Document your metric selection rationale in your technical documentation — the "why" matters as much as the "what" during conformity assessment.

Practical bias testing workflow

The following seven-step workflow covers the full lifecycle of bias testing for an Article 10-compliant data governance process:

Step 1: Define protected groups

Identify which protected attributes are relevant to your system's context. Not every attribute applies to every system — a medical device has different relevant attributes than a recruitment tool. Start with the protected attributes table above and filter based on:

• The domain of deployment (Annex III area — see the high-risk classification guide)
• The input features your model uses (including potential proxy variables)
• The population the system will serve
• Known historical discrimination patterns in your domain

Document this selection and its rationale. Conformity assessors will want to see why you tested for certain attributes and not others.

Step 2: Collect and annotate demographic data

This is where Article 10(5) becomes critical. To test for bias, you need demographic annotations — but collecting and processing this data triggers GDPR special category protections.

Practical approaches:

• Direct collection with consent: Survey participants or data subjects. Gold standard but often impractical at scale.
• Statistical inference using public reference data: Compare model outcome distributions against known demographic distributions from census data. Avoids processing individual-level special category data.
• Proxy-based estimation: Use known correlations (e.g., first-name-to-gender mapping, postcode-to-ethnicity distributions) with uncertainty quantification. Document the methodology and limitations.
• Synthetic data augmentation: Generate synthetic demographic annotations using differential privacy techniques to test bias without exposing real individual data.

Apply Article 10(5) safeguards to all methods: pseudonymisation or encryption, strict access controls, technical limitations on re-use, and documented legal basis.

Step 3: Select appropriate fairness metrics

Choose your metrics based on the decision in the fairness metrics section above. Select at minimum:

• One group-level outcome metric (demographic parity or predictive parity)
• One group-level error metric (equalized odds or equal opportunity)
• One calibration metric (if the system outputs scores or probabilities)

Define acceptable thresholds. There is no universal standard, but common practice is to flag disparities where the ratio between the least-favoured and most-favoured group drops below 0.8 (the "four-fifths rule" originating from US employment discrimination law, widely used in EU practice as a reasonable threshold).

Step 4: Run statistical tests on training, validation, and test data

Before evaluating model behaviour, examine the data itself for bias signals:

• Composition analysis: Are protected groups represented proportionally to the deployment population?
• Label distribution analysis: Do label rates differ systematically across groups? Is this justified or reflective of historical bias?
• Feature distribution analysis: Do feature distributions differ across groups in ways irrelevant to the prediction task?
• Missing data analysis: Is data missingness correlated with protected group membership?

Use chi-squared tests for categorical features, Kolmogorov-Smirnov tests for continuous features, and disparity ratio calculations for labels. Document every finding.

Step 5: Run model-level fairness evaluation

With the model trained, evaluate fairness on the held-out test set:

1. Compute chosen fairness metrics disaggregated by each protected attribute.
2. Compute intersectional metrics — test at the intersection of multiple attributes (e.g., young women of ethnic minority background). Single-axis testing misses intersectional disparities.
3. Run counterfactual fairness tests — change a data point's protected attribute while keeping other features constant. Large prediction changes indicate direct discrimination.
4. Evaluate across decision thresholds — a system that appears fair at one threshold may be unfair at another.
5. Perform error analysis by subgroup — equal overall accuracy can mask dramatically different error patterns across groups.

Step 6: Document findings and remediation

Every finding must be documented in your Annex IV technical documentation. For each bias issue identified:

• Describe the bias: What type, which groups affected, what magnitude.
• Assess the impact: What fundamental rights are at risk? Reference specific EU Charter articles. This feeds directly into the FRIA if applicable.
• Document remediation: What measures did you take? Options include:
  • Pre-processing: Rebalancing datasets, reweighting samples, removing proxy features
  • In-processing: Fairness constraints during training, adversarial debiasing, regularisation
  • Post-processing: Threshold adjustment per group, outcome calibration, rejection option classification
• Document residual risk: If bias cannot be fully mitigated, what residual disparities remain and why are they acceptable? What compensating controls (e.g., human oversight) are in place?

Step 7: Monitor in production

Bias is not static. Data distributions shift, populations change, user behaviour evolves, and upstream data sources are modified. Your risk management system must include ongoing bias monitoring:

• Outcome monitoring: Track decision distributions across protected groups. Set automated alerts for threshold breaches.
• Performance monitoring: Track disaggregated accuracy, precision, recall, and calibration using ground truth labels when available.
• Drift detection: Monitor for distribution shifts in input features that correlate with protected attributes.
• Feedback loop analysis: If outputs influence future training data (e.g., approved loans become positive labels), monitor for self-reinforcing bias loops.
• Periodic re-evaluation: Full bias testing at minimum quarterly for high-risk systems, and triggered by any significant model, data, or deployment change.

Open-source bias testing tools

You do not need to build bias testing infrastructure from scratch. Several mature, well-maintained open-source libraries implement the metrics and workflows described above.

Integration recommendation: For most teams building Article 6 high-risk AI systems, start with Fairlearn for its balance of simplicity and capability. Add AIF360 for specialised metrics. Use the What-If Tool for stakeholder communication. All tools produce outputs compatible with Annex IV documentation requirements.

Real-world bias testing scenarios

Scenario 1: Credit scoring AI — gender and ethnic bias

A fintech company provides an AI credit scoring system to EU retail banks — Annex III point 5(a), high-risk under Article 6.

Bias testing approach:

1. Protected groups tested: Gender, ethnic origin (inferred from postcode-level census data using Article 10(5) safeguards), age bands.
2. Fairness metrics applied: Equalized odds and calibration (a 650 score should mean the same default probability regardless of group).
3. Findings: False rejection rate for women was 1.4x higher than for men — traced to historical bias in training data from a period when women had fewer independent credit histories. Calibration accuracy was lower for ethnically diverse postcodes due to representation bias.
4. Remediation: Reweighted training samples, applied stratified sampling for geographic balance, implemented Fairlearn threshold optimiser. Residual disparity reduced to 1.05x.
5. Documentation: Full methodology recorded in Annex IV Sections 2 and 3. FRIA findings communicated to deploying banks.

Scenario 2: HR screening tool — age and disability discrimination

A SaaS company offers an AI CV screening tool — Annex III point 4(a), high-risk.

Bias testing approach:

1. Protected groups tested: Age (graduation year proxy), disability (gap patterns), gender (first name inference), ethnic origin (name-ethnicity models).
2. Fairness metrics applied: Demographic parity and equal opportunity.
3. Findings: Candidates over 45 shortlisted at 0.6x the rate of 25-35 year-olds — below the four-fifths threshold. The model penalised career gaps and valued "recent" certifications, both age-correlated. Disability-related employment gaps reduced scores by 18% via the "months of continuous employment" proxy.
4. Remediation: Removed "years since last certification." Replaced "months of continuous employment" with "total months of relevant experience." Applied adversarial debiasing. Demographic parity ratio improved from 0.60 to 0.83.
5. Documentation: Published in technical documentation. Deployers informed of residual disparities and human oversight requirements per HR compliance obligations.

Scenario 3: Healthcare AI — demographic representation gaps

A medical device company develops an AI skin cancer detection system from dermoscopic images — Annex III point 5(c), high-risk.

Bias testing approach:

1. Protected groups tested: Skin type (Fitzpatrick scale I-VI), gender, age groups (particularly geriatric populations).
2. Fairness metrics applied: Equal opportunity (sensitivity must be equal across skin types) and calibration.
3. Findings: Sensitivity for Fitzpatrick Type I-III was 94% but dropped to 79% for Type IV-VI. Root cause: 82% of training images were lighter-skinned patients. Melanoma presents differently on darker skin (acral locations, different colour patterns) and the model lacked sufficient examples. Sensitivity for patients aged 75+ was 12 points lower than for 40-60 year-olds due to comorbid conditions.
4. Remediation: Partnered with clinics in Sub-Saharan Africa and South Asia for 15,000 additional dark skin images. Applied augmentation and implemented stratified evaluation requiring minimum 90% per-group sensitivity. Post-remediation Type IV-VI sensitivity improved to 91%.
5. Documentation: Annex IV includes explicit performance breakdowns by Fitzpatrick type and age. The risk management system includes quarterly sensitivity monitoring by skin type.

Documentation requirements

Article 11 and Annex IV impose specific documentation requirements that directly relate to bias testing. Your technical documentation must include:

Key principle: Document not just what you did, but why you chose that approach over alternatives. A methodology that acknowledges limitations is more credible than one that claims perfect fairness.

For all Annex IV sections, see the technical documentation template guide. For the broader picture, refer to the EU AI Act compliance checklist.

Is bias testing mandatory for all AI systems?

No. Bias testing under Article 10 is mandatory only for high-risk AI systems classified under Article 6 and listed in Annex III. However, GDPR Article 22 (automated decision-making) and EU non-discrimination directives create independent legal bases for fairness testing regardless of AI Act classification. Use the AI Act risk assessment tool to determine whether your system is high-risk.

Can I use synthetic data instead of real demographic data for bias testing?

Partially. Synthetic data can supplement bias testing — particularly for underrepresented groups — but cannot fully replace real-world data testing. Synthetic data may not capture complex correlations and edge cases present in real distributions. Use real data with Article 10(5) safeguards for primary evaluation, synthetic data for stress-testing and augmentation, and document the methodology and its limitations.

What happens if my system fails bias testing?

A bias test failure is not a compliance failure — it is information. The obligation is to examine, detect, and mitigate, not to achieve zero bias. If you discover bias, you must: (1) document the finding, (2) assess severity, (3) implement remediation, (4) document residual disparity, and (5) implement compensating controls such as human oversight. Discovering bias and taking no action — that is a compliance failure.

How often must bias testing be repeated?

The AI Act does not specify a fixed frequency. Article 9 requires risk management to be a "continuous iterative process." In practice: (1) full bias testing before initial deployment, (2) re-evaluation after any significant model update, (3) re-evaluation when deployment context changes, (4) periodic re-evaluation — quarterly is common for high-risk systems, and (5) triggered re-evaluation when monitoring detects distributional shifts.

Does Article 10(5) override GDPR restrictions on processing sensitive data?

Article 10(5) does not override GDPR — it provides a specific, additional legal basis for processing special category data strictly for bias monitoring, detection, and correction. You must still comply with GDPR principles: data minimisation, purpose limitation, storage limitation, and security. The processing must be strictly necessary, accompanied by appropriate safeguards (pseudonymisation, encryption, access controls), and limited to the bias testing purpose. Consult your DPO and document your legal basis analysis.

Can I outsource bias testing to a third party?

Yes, but the provider retains legal responsibility. You must: (1) ensure the third party has appropriate expertise, (2) define scope and methodology in advance, (3) review and validate findings, (4) integrate into Annex IV documentation, and (5) retain the ability to reproduce the analysis. Outsourcing the work does not outsource the obligation.

Need help implementing AI bias testing for your high-risk AI system? Take the Legalithm AI Act assessment to understand your compliance obligations and get a tailored action plan.