Guides the combined DPIA and AI Act conformity assessment for AI systems processing personal data. Covers EDPB-EDPS Joint Opinion 5/2021, training data lawfulness under Art. 6 and Art. 9, Art. 22 automated decision-making, algorithmic bias detection, and NIST AI RMF MAP function. Keywords: AI privacy, DPIA, AI Act, algorithmic bias, automated decision-making, Art. 22, training data, NIST AI RMF.
Instrucciones de origen · Vista previa de solo lectura
name
ai-privacy-assessment
title
Conducting AI System Privacy Assessment
description
Guides the combined DPIA and AI Act conformity assessment for AI systems processing personal data. Covers EDPB-EDPS Joint Opinion 5/2021, training data lawfulness under Art. 6 and Art. 9, Art. 22 automated decision-making, algorithmic bias detection, and NIST AI RMF MAP function. Keywords: AI privacy, DPIA, AI Act, algorithmic bias, automated decision-making, Art. 22, training data, NIST AI RMF.
AI systems that process personal data require a combined privacy and conformity assessment addressing both GDPR obligations and the EU AI Act (Regulation 2024/1689). This skill integrates the GDPR Art. 35 DPIA framework with AI-specific risk assessment, encompassing training data lawfulness, Art. 22 automated decision-making implications, algorithmic fairness, and the NIST AI Risk Management Framework MAP function. The assessment methodology draws from the EDPB-EDPS Joint Opinion 5/2021 on the AI Act proposal and subsequent EDPB Guidelines 06/2025 on AI and data protection.
Legal Framework
GDPR Provisions Applicable to AI
Provision
Application to AI Systems
Art. 5(1)(a) — Lawfulness, fairness, transparency
AI processing must have a lawful basis; the logic of AI decisions must be explainable to data subjects
Art. 5(1)(b) — Purpose limitation
Training data collected for one purpose cannot be used to train AI models for an incompatible purpose without further lawful basis
Art. 5(1)(c) — Data minimisation
AI models should not require more personal data than necessary; synthetic data and anonymisation should be considered
Art. 5(1)(d) — Accuracy
AI outputs affecting individuals must be accurate; model drift must be monitored
Art. 6(1) — Lawful basis
Each stage of AI processing (data collection, model training, inference, output use) requires a lawful basis
Art. 9 — Special categories
Training on health, biometric, genetic, racial, political, religious, sexual orientation, or trade union data requires an Art. 9(2) exemption
Art. 13-14 — Transparency
Privacy notices must disclose the existence of automated decision-making, meaningful information about the logic involved, and the significance and envisaged consequences
Art. 22 — Automated decision-making
Data subjects have the right not to be subject to decisions based solely on automated processing that produce legal effects or similarly significantly affect them, with exceptions under Art. 22(2)
Art. 25 — Data protection by design
AI systems must embed privacy protections from the design phase: privacy-preserving ML techniques, differential privacy, federated learning
Art. 35 — DPIA
AI systems meeting EDPB WP248rev.01 criteria (evaluation/scoring, automated decision-making, innovative technology) require a DPIA
EU AI Act (Regulation 2024/1689) Risk Classification
Risk Category
Description
AI Act Requirements
GDPR Overlap
Unacceptable Risk (Art. 5)
AI practices prohibited outright: social scoring by public authorities, real-time remote biometric identification in public spaces (with exceptions), emotion recognition in workplace/education, untargeted scraping for facial recognition databases
Prohibited — cannot be deployed
Art. 9 special categories, Art. 35 DPIA
High Risk (Annex III)
AI systems in listed areas: biometrics, critical infrastructure, education, employment, essential services, law enforcement, migration, administration of justice
Conformity assessment, risk management system, data governance, transparency, human oversight, accuracy/robustness/cybersecurity
Art. 22, Art. 35 DPIA, Art. 25 DPbD
Limited Risk (Art. 50)
AI systems with specific transparency obligations: chatbots, emotion recognition, deep fakes
Transparency obligations (notify users they are interacting with AI)
Art. 13-14 transparency
Minimal Risk
All other AI systems
No specific AI Act obligations; voluntary codes of practice
Standard GDPR compliance
EDPB-EDPS Joint Opinion 5/2021
Key recommendations from the Joint Opinion on the AI Act proposal:
Prohibition of remote biometric identification in public spaces — supported without exceptions
AI-specific DPIA — the EDPB recommended mandatory DPIAs for all high-risk AI systems, not limited to Art. 35 triggers
Right to explanation — Art. 22(3) right to obtain human intervention and contest decisions should apply to all AI-assisted decisions, not only fully automated ones
Training data governance — controllers must ensure lawfulness of personal data used for AI training throughout the model lifecycle, including when training data is subsequently deleted
Purpose limitation for AI training — repurposing personal data for AI model training requires a compatibility assessment under Art. 6(4) or a separate lawful basis
Assessment Methodology
Phase 1: AI System Classification (Week 1)
1.1 AI Act Risk Classification
Determine the AI system's risk category under the AI Act:
Check Art. 5 prohibited practices list — if the system falls within a prohibition, it cannot be deployed.
Check Annex III high-risk categories:
Biometric identification and categorisation of natural persons
Management and operation of critical infrastructure
Education and vocational training (admissions, assessment, proctoring)
Employment, worker management, and access to self-employment (recruitment, task allocation, performance evaluation, termination)
Access to and enjoyment of essential private and public services (credit scoring, insurance pricing, emergency services dispatch)
Law enforcement (crime risk assessment, polygraph, evidence evaluation)
Migration, asylum, and border control
Administration of justice and democratic processes
Check Art. 50 transparency obligations for limited-risk AI.
If not high-risk or limited-risk, classify as minimal risk.
1.2 GDPR Processing Assessment
Identify all personal data processed at each AI lifecycle stage: data collection, data preparation, model training, model validation, inference/deployment, output use.
For each stage, identify the lawful basis under Art. 6(1).
If special category data is involved, identify the Art. 9(2) condition.
Assess whether Art. 22 applies (solely automated decisions with legal or significant effect).
Phase 2: Training Data Lawfulness (Week 2)
2.1 Data Collection Lawfulness
Assessment Question
Requirement
Was the training data collected with a lawful basis under Art. 6(1)?
Each data source must have an identified lawful basis
Is the use of data for AI training compatible with the original collection purpose?
Art. 6(4) compatibility assessment or Art. 5(1)(b) further processing analysis
Was consent obtained for AI training specifically?
If relying on Art. 6(1)(a), consent must be specific, informed, and freely given for the AI training purpose
If using legitimate interest, has a balancing test been conducted?
Art. 6(1)(f) requires documented LIA including AI-specific impacts
Does training data include special categories?
Art. 9(2) exemption required; Art. 9(2)(j) scientific research may apply with safeguards
2.2 Data Quality and Bias Assessment
Assessment Area
Requirements
Representativeness
Training data must be representative of the population the AI system will be applied to. Underrepresentation of demographic groups must be identified and addressed.
Label accuracy
If supervised learning, labels must be accurate and free from historical bias. Human labellers must be trained on anti-discrimination principles.
Temporal validity
Training data must reflect current conditions. Stale training data can produce discriminatory outputs.
Proxy variables
Identify features that may serve as proxies for protected characteristics (postcode as proxy for ethnicity, name as proxy for gender).
Data provenance
Document the source, collection methodology, and processing history for all training data.
2.3 Training Data Retention
Art. 5(1)(e) storage limitation applies to training data.
Once training is complete, assess whether original training data must be retained or can be deleted.
If training data is deleted, document how data subject rights (access, erasure) will be facilitated regarding data embedded in the trained model.
Consider whether the trained model itself constitutes personal data (if individual data points can be extracted through model inversion attacks).
Does the AI system make decisions about individuals?
├─ NO → Art. 22 does not apply.
└─ YES → Continue.
│
Are decisions based solely on automated processing?
├─ NO → Art. 22(1) does not apply, but Art. 13-14 transparency still applies.
│ (Note: "meaningful human involvement" must be genuine, not rubber-stamping.)
└─ YES → Continue.
│
Do decisions produce legal effects or similarly significantly affect the individual?
├─ NO → Art. 22(1) does not apply.
└─ YES → Art. 22(1) applies. The individual has the right not to be subject
to the decision unless an Art. 22(2) exception applies.
3.2 Art. 22(2) Exceptions
Exception
Requirements
Art. 22(2)(a) — Necessary for contract
Decision must be necessary for entering into or performing a contract with the data subject
Art. 22(2)(b) — Authorised by law
Union or Member State law must authorise the decision and provide suitable safeguards
Art. 22(2)(c) — Explicit consent
Data subject has given explicit consent to the automated decision
3.3 Art. 22(3) Safeguards
Even where an Art. 22(2) exception applies, the controller must implement suitable measures including:
Right to obtain human intervention
Right to express the data subject's point of view
Right to contest the decision
Meaningful information about the logic involved (not the full algorithm, but sufficient to understand the factors and their relative weight)
Phase 4: Algorithmic Bias and Fairness Assessment (Week 3-4)
4.1 Fairness Metrics
Metric
Description
Application
Demographic parity
Positive outcome rates should be equal across protected groups
Credit scoring, hiring
Equalized odds
True positive and false positive rates should be equal across groups
Criminal risk assessment, fraud detection
Predictive parity
Positive predictive value should be equal across groups
Medical diagnosis, recidivism prediction
Individual fairness
Similar individuals should receive similar outcomes
Loan pricing, insurance premium calculation
Counterfactual fairness
Outcome should not change if only the protected characteristic changes
Any decision-making system
4.2 Bias Testing Protocol
Define protected characteristics relevant to the jurisdiction and use case (race, gender, age, disability, religion, sexual orientation per Equality Act 2010 / EU Charter Art. 21).
Partition test dataset by protected characteristics.
Calculate fairness metrics for each group.
Identify disparate impact: if the selection rate for any protected group is less than 80% of the rate for the group with the highest rate (four-fifths rule from US EEOC, widely adopted as a starting benchmark).
If disparate impact is detected, assess whether the disparity is justified by a legitimate, non-discriminatory factor.
Document bias testing methodology, results, and remediation actions.
Phase 5: NIST AI RMF MAP Function Integration (Week 4)
The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) MAP function identifies the context, capabilities, and potential impacts of AI systems. Integrate the following MAP subcategories:
Subcategory
Assessment Action
MAP 1.1
Document the intended purpose, context of use, and deployment environment
MAP 1.2
Document interdependent and interconnected systems
MAP 1.5
Identify intended users, affected individuals, and stakeholders
MAP 1.6
Assess impacts on individuals, groups, communities, organisations, and society
MAP 2.1
Establish the AI system's knowledge limits and conditions where it may fail
MAP 2.2
Document scientific integrity of training and testing methodologies
MAP 2.3
Assess environmental impact of AI system training and deployment
MAP 3.1
Document potential benefits of the AI system
MAP 3.2
Document potential costs, risks, and negative impacts
MAP 3.4
Map risks specifically to affected communities and stakeholders
MAP 3.5
Document likelihood and severity of identified risks
MAP 5.1
Engage with diverse stakeholders and affected communities
MAP 5.2
Engage with domain experts, AI practitioners, and sociotechnical experts
Phase 6: Combined Risk Assessment and Reporting (Week 5-6)
Consolidate GDPR DPIA risks, AI Act compliance gaps, and NIST AI RMF findings.
For each risk, assess likelihood and severity using the DPIA risk matrix.
Identify mitigation measures combining privacy controls and AI governance controls.
Calculate residual risk levels.
If high-risk AI under the AI Act, prepare conformity assessment documentation.
Submit to DPO for Art. 35(2) advice.
If residual risk remains high, initiate Art. 36 prior consultation.
AI-Specific Privacy Risks
Risk Category
Description
Mitigation Approach
Model inversion
Attacker reconstructs training data from model outputs
Differential privacy during training, output perturbation, access controls on model API
Membership inference
Attacker determines whether a specific individual's data was in the training set
Regularisation, differential privacy, limiting model confidence scores
Data poisoning
Malicious manipulation of training data to bias model outputs
Training data provenance verification, anomaly detection, robust training techniques
Concept drift
Model accuracy degrades over time as real-world data distribution changes
Continuous monitoring, automated retraining triggers, human review of edge cases
Explanation manipulation
Gaming of AI explanations to hide discriminatory factors
Multiple explanation methods, adversarial testing of explanation consistency
Feedback loops
AI decisions create data that reinforces existing biases
Regular bias auditing, human-in-the-loop review, outcome monitoring by demographic group
Common Assessment Deficiencies
Treating AI training as a single processing operation: Each stage (collection, preparation, training, inference) is a distinct processing operation requiring its own lawful basis assessment.
Relying on anonymisation claims without verification: Data claimed to be anonymised may be re-identifiable through model inversion or linkage attacks.
Inadequate Art. 22 human involvement: Rubber-stamp human review of AI decisions does not constitute meaningful human involvement.
Ignoring purpose limitation for training data: Using customer data collected for service delivery to train AI models without compatibility assessment.
No bias testing on protected characteristics: Fairness assessment limited to overall accuracy without demographic disaggregation.
Static assessment: AI systems evolve through retraining; the assessment must be updated when the model is retrained or the deployment context changes.
Enforcement Precedents
Italian Garante vs Clearview AI (2022): EUR 20 million fine for scraping biometric data from the internet to train facial recognition AI without lawful basis, consent, or DPIA.
Italian Garante vs Deliveroo (2021): EUR 2.5 million fine for algorithmic management of food delivery riders constituting Art. 22 automated decision-making without adequate safeguards. The algorithm assigned reputation scores and allocated delivery shifts without meaningful human review.
Hungarian DPA vs Budapest Bank (2023): Fine for automated credit scoring system that used postcode as a proxy for ethnicity without bias impact assessment.
CNIL vs Clearview AI (2022): EUR 20 million fine; CNIL ordered deletion of all data of French residents and prohibited further collection.
Dutch DPA vs Tax Authority (2020): EUR 3.7 million fine for algorithmic profiling system (SyRI) that disproportionately targeted ethnic minorities in fraud detection, violating ECHR Art. 8 and Art. 14.
