| name | audit-training-data-integrity |
| description | Use when curating, collecting, or managing datasets for training or fine-tuning machine learning models — to detect and mitigate poisoned, biased, mislabeled, or adversarially crafted training data. |
| source | OWASP Top 10 for LLM Applications 2025 LLM03 (owasp.org/www-project-top-10-for-large-language-model-applications/); NIST AI RMF 1.0; Barreno et al. "The Security of Machine Learning" (2010); CWE-20 |
| tags | ["security","owasp","llm","training-data","data-poisoning","ml-security","emerging","developer"] |
| emerging | true |
Audit Training Data Integrity
Validate, deduplicate, and provenance-track training data to detect poisoning attacks, privacy violations, and systematic bias before they are baked into model weights.
Why This Is Best Practice
Adopted by: OWASP Top 10 for LLM Applications 2025 LLM03 (Training Data Poisoning). NIST AI RMF 1.0 (2023) Govern 1.1 and Map 2.2 address training data governance. Google, Meta, and Hugging Face publish data cards and datasheets for datasets as standard practice. The ML safety community (papers by Carlini, Wallace, Schuster) has demonstrated practical data poisoning attacks since 2020.
Status: Emerging — data poisoning is a well-researched attack (since ~2017) but detection tooling and industry standards are still maturing.
Impact: Carlini et al. (2021) demonstrated extracting verbatim training data from GPT-2, including PII. Schuster et al. (2021) showed code suggestion models trained on poisoned GitHub data could insert security vulnerabilities into suggestions (later confirmed in real systems). BadNL attacks (Chen et al., 2021) showed NLP classifiers could be given hidden backdoors via 50 poisoned examples in 100,000 training samples — 0.05% poisoning rate sufficient for reliable triggering.
Why best: Post-hoc model auditing (testing the trained model for backdoors) is the alternative — it's expensive, slow, and incomplete. Data-time auditing catches issues before they're encoded into weights, where removal is computationally expensive and may require full retraining.
Sources: OWASP LLM Top 10 2025 LLM03; NIST AI RMF 1.0; Carlini et al. "Extracting Training Data from Large Language Models" (2021); Schuster et al. "You Autocomplete Me: Poisoning Vulnerabilities in Neural Code Completion" (2021)
Steps
-
Track data provenance with a data card for every dataset:
dataset_name: customer-support-v2
version: 2.3.1
collected_by: data-team
collection_date: 2024-01-15
sources:
- type: internal
description: Customer support tickets Q3-Q4 2023
pii_review: completed
pii_review_date: 2024-01-10
licenses: [internal-proprietary]
known_biases:
- English-only, over-represents US timezone users
exclusions:
- removed: all tickets containing SSN, credit card numbers
- removed: tickets from users who opted out of data use
Rules
- Cryptographically hash and version-stamp each training dataset snapshot — enables auditing exactly which data was used if a model issue is discovered post-deployment.
- Data contributed by users requires explicit consent for training use — check privacy policy and applicable regulations (GDPR Article 5, CCPA).
- Regular expressions alone are insufficient for PII detection — use NER-based tools (Presidio, spaCy) for higher coverage.
- Poisoning attacks are designed to be undetectable in individual samples — statistical methods on the full dataset are required.
Common Mistakes
- Treating training data as inherently trusted because it's "internal" — internal datasets sourced from user content or web scrapes can contain adversarial examples.
- Not tracking data provenance — when a model exhibits a problem behavior, inability to trace which training data caused it makes remediation impossible.
- Using the full training set for deduplication checks — exact deduplication misses near-duplicates; use MinHash LSH or similar approximate methods.
- Skipping PII scanning because data is "anonymized" — anonymization is often incomplete; structured PII like SSNs and credit cards persist through naive scrubbing.
