audit-llm-data-disclosure

name	audit-llm-data-disclosure
description	Use when deploying an LLM application that may have been trained on or given access to sensitive data — to assess the risk of the model leaking PII, proprietary information, or confidential context through its outputs.
source	OWASP Top 10 for LLM Applications 2025 LLM06 (owasp.org/www-project-top-10-for-large-language-model-applications/); Carlini et al. "Extracting Training Data" (2021); NIST AI RMF 1.0; CWE-312
tags	["security","owasp","llm","data-disclosure","pii","privacy","ai-security","emerging"]
emerging	true

Audit LLM Data Disclosure

Assess and mitigate the risk of LLMs revealing sensitive information — through training data extraction, system prompt leakage, or context window exposure — using probing tests, output filtering, and architectural separation.

Why This Is Best Practice

Adopted by: OWASP Top 10 for LLM Applications 2025 LLM06 (Sensitive Information Disclosure). NIST AI RMF 1.0 (2023) Map 2.1 includes privacy-related risks in AI systems. The EU AI Act (2024) Article 10 requires training data governance for high-risk AI. Google, Apple, Samsung, and JPMorgan have all issued employee AI usage policies specifically to prevent sensitive data disclosure through LLMs. Status: Emerging — training data extraction has been demonstrated in peer-reviewed research since 2021, but practical mitigations are still being developed. Impact: Carlini et al. (2021) extracted verbatim PII (names, email addresses, phone numbers) from GPT-2 with targeted prompting. Samsung engineers accidentally submitted proprietary chip schematics to ChatGPT (2023) — stored in OpenAI's training pipeline. Microsoft GitHub Copilot has been demonstrated to reproduce copyrighted code verbatim. A system prompt containing an API key returned to users when prompted correctly can expose production credentials. Why best: Reactive filtering (scanning all LLM outputs for PII patterns) is the alternative — it misses information that was encoded differently in the training data or context. Architectural separation (not putting sensitive data in training data or the context window) is the primary defense; output scanning is defense-in-depth.

Sources: OWASP LLM Top 10 2025 LLM06; Carlini et al. "Extracting Training Data from Large Language Models" (2021); Samsung data leak (2023); NIST AI RMF 1.0

Steps

1. Probe for training data memorization before deployment:

   def probe_for_memorization(model, sensitive_prefixes):
       """
       Test if the model completes known sensitive prefixes verbatim.
       Use prefixes from data you know was in training — if it completes
       accurately, memorization has occurred.
       """
       findings = []
       for prefix, expected_completion, sensitivity_label in sensitive_prefixes:
           completion = model.generate(prefix, max_tokens=50)
           if expected_completion.lower() in completion.lower():
               findings.append({
                   'prefix': prefix[:50] + '...',
                   'sensitivity': sensitivity_label,
                   'risk': 'HIGH'
               })
       return findings
   
   # Test with sanitized prefixes from your training data
   test_cases = [
       ("The API key for production is", "sk-prod-...", "api_key"),
       ("John Smith's SSN is", "123-45-", "pii"),
   ]
   

2. Scan LLM outputs for PII and sensitive patterns before returning to users:

   import re
   from presidio_analyzer import AnalyzerEngine
   
   analyzer = AnalyzerEngine()
   
   PII_PATTERNS = [
       (r'\b\d{3}-\d{2}-\d{4}\b', 'SSN'),
       (r'\b4[0-9]{12}(?:[0-9]{3})?\b', 'Visa card'),
       (r'\b[A-Za-z0-9._%+-]+@[A-Za-z0-9.-]+\.[A-Z|a-z]{2,}\b', 'email'),
       (r'\b(?:sk-|pk_|sk_live_)[A-Za-z0-9]{20,}\b', 'API key pattern'),
   ]
   
   def () -> :
       findings = []
       
       presidio_results = analyzer.analyze(text=text, language=)
       findings.extend([r  r  presidio_results  r.score > ])
       
        pattern, label  PII_PATTERNS:
            re.search(pattern, text):
               findings.append({: label, : })
        findings
   
    () -> :
       findings = scan_output_for_sensitive_data(raw_output)
        findings:
           logger.warning(, extra={: findings})
           
            redact_sensitive_patterns(raw_output)
        raw_output
   

Rules

• System prompt confidentiality is not a reliable security control — treat system prompt contents as potentially disclosable.
• Never put production credentials, private keys, or HIPAA/PCI-protected data in LLM context windows.
• Output scanning catches structured PII (SSNs, card numbers) reliably but misses semantic disclosure (paraphrased personal information) — architectural prevention is required.
• Fine-tuned or RAG-augmented models have higher memorization risk than base models with in-context retrieval.

Common Mistakes

• Assuming the model "won't reveal" confidential information because it was told not to — this is a policy, not a technical control.
• Not scanning outputs in RAG systems — retrieved documents containing PII may be reproduced verbatim by the LLM in its response.
• Logging full LLM context windows — creates a secondary data store containing all the sensitive information sent to the model.
• Testing only direct extraction prompts — indirect extraction (asking the model to "play a game" involving sensitive data) bypasses direct refusal logic.