synthetic-data-security

name	synthetic-data-security
description	Review synthetic training data generation pipelines for distillation attack risks, model collapse vectors, privacy leakage through synthetic outputs, and quality controls that prevent degraded or adversarially biased synthetic data from entering training pipelines.
last_reviewed	"2026-04-30T00:00:00.000Z"

Synthetic Data Security

First Principle

Synthetic data inherits the biases, errors, and vulnerabilities of the model that generated it. Train on it uncritically and those properties compound.

Synthetic data generated by LLMs is increasingly used to augment or replace human-labeled data. This introduces a feedback loop: a model trained on synthetic data generated by a predecessor model can amplify predecessor errors, drift from real-world distributions, and — in adversarial contexts — be manipulated through the generator model to implant behaviors in the successor. The quality and security of synthetic data depends entirely on the integrity of the generation pipeline.

Attack Mental Model

1. Distillation attack — an adversary uses a target model's API to generate synthetic training data that encodes the target model's behavior, then trains a derivative model that reproduces the target's capabilities in violation of terms of service or without authorization.
2. Generator poisoning — if the model used to generate synthetic data has been compromised (via prompt injection or fine-tuning), it generates synthetic examples that introduce backdoors or behavioral biases into models trained on its output.
3. Model collapse via feedback loops — synthetic data generated from a model is used to train a new version of that model. Without real-data anchoring, successive generations lose diversity and drift toward degenerate distributions — degrading capability and safety properties.
4. Privacy amplification — a generator model that memorized sensitive training data can reproduce it in synthetic outputs. Synthetic datasets that appear privacy-safe may contain near-verbatim reproductions of PII or proprietary content from the generator's training set.

Control Lens

Principle	What It Means Here
Validate	Synthetic data is validated for quality, diversity, and absence of privacy-sensitive content before inclusion in training corpora.
Scope	Synthetic data generated by a teacher model is clearly labeled with its source model and generation parameters. It does not silently enter training pipelines as if it were human-generated.
Isolate	Feedback loops between generator model versions are broken by anchoring each training run with a minimum fraction of verified real-world data.
Enforce	Generation pipelines are access-controlled so that unauthorized use of production models to generate competitor training data is detected and blocked.

SDS.1 Synthetic Data Quality and Privacy Controls

The core vulnerability: Synthetic data that reproduces memorized training content from the generator model introduces privacy violations into successor model training corpora — often without any indication in the generated text that it is reproduced content.

Check

• Is generated synthetic data scanned for near-verbatim reproductions of known sensitive content (PII, proprietary documents, personal data)?
• Is diversity measured before synthetic data is added to training? Homogeneous synthetic data accelerates model collapse.
• Are generation parameters (temperature, top-p, model version, prompt template) recorded alongside synthetic examples for reproducibility and audit?

Action

• Scan synthetic data for memorized content before inclusion:

from difflib import SequenceMatcher

SENSITIVE_CORPUS = load_sensitive_documents()  # PII, proprietary content

def scan_for_memorization(synthetic_text: str, threshold: float = 0.85) -> bool:
    for reference in SENSITIVE_CORPUS:
        ratio = SequenceMatcher(None, synthetic_text, reference).ratio()
        if ratio > threshold:
            log_security_event("synthetic_memorization_detected", ratio=ratio)
            return True  # reject this example
    return False

def compute_diversity(synthetic_batch: list[str]) -> float:
    # Self-BLEU: lower = more diverse
    return compute_self_bleu(synthetic_batch)

• Record generation provenance for every synthetic example:

@dataclass
class SyntheticExample:
    content: str
    label: str | None
    generator_model_id: str
    generator_model_version: str
    generation_temperature: float
    generation_prompt_hash: str  # hash of the prompt template used
    generation_timestamp: str
    human_reviewed: bool

• Enforce minimum diversity thresholds. A synthetic batch with self-BLEU > 0.7 is too homogeneous — reject or regenerate with higher temperature. Homogeneous synthetic data causes rapid model collapse in successive training iterations.

Failure Modes

• A synthetic dataset generated by a large model contains near-verbatim reproductions of PII from the generator's training set. The synthetic dataset is treated as privacy-safe because no human labeled PII into it. The successor model inherits and reproduces the PII.
• A synthetic data generation pipeline runs at temperature=0.1 for consistency. All generated examples are nearly identical in structure. Successive training runs on this data collapse the model's output diversity.

SDS.2 Distillation Prevention and Feedback Loop Control

The core vulnerability: An API that returns high-quality outputs at scale can be systematically mined to create a synthetic training set that distills the model's capabilities — potentially in violation of terms of service or at the cost of the API provider subsidizing a competitor's training.

Check

• Is systematic synthetic data generation via the API detectable from usage patterns — high volume, high diversity of prompts, consistent low latency?
• Is there a feedback loop where model N+1 is trained on data generated by model N? If so, is there a minimum real-data fraction requirement to prevent model collapse?
• Are terms of service restrictions on training data generation enforced technically (not only contractually)?

Action

• Detect distillation-pattern API usage:

def score_distillation_pattern(session: APISession) -> float:
    return (
        session.unique_prompt_ratio * 0.4         # high unique prompts = coverage seeking
        + (1 - session.output_diversity_score) * 0.3  # low output diversity = label collection
        + min(session.sustained_rpm / 100, 1.0) * 0.3  # sustained high RPM = bulk collection
    )

• Enforce real-data anchoring in training pipelines. Any training run using synthetic data must include at minimum 20% verified real-world data. This breaks runaway feedback loops that cause model collapse.
• Version and isolate synthetic data by generator model. When a generator model is updated or deprecated, synthetic data generated by it is archived and not used in new training runs without re-evaluation. This prevents "ghost teacher" influence on future generations.

Minimum Deliverable Per Review

• Memorization scan: near-verbatim reproduction detection against sensitive content before synthetic data inclusion
• Diversity measurement: self-BLEU or equivalent computed per synthetic batch; homogeneous batches rejected
• Generation provenance: model ID, version, temperature, prompt hash recorded per synthetic example
• Real-data anchoring: minimum real-data fraction enforced in training runs that use synthetic data
• Feedback loop versioning: synthetic data tagged by generator version; not silently reused across generations
• Distillation detection: API usage monitoring for systematic synthetic dataset collection patterns

Quick Win

Add a memorization scan to your synthetic data pipeline. Before adding any batch of generated data to a training corpus, run a similarity check against your known sensitive documents. Near-verbatim matches (>85% sequence similarity) should be discarded. This costs one similarity computation per generated example and eliminates the most direct privacy risk.

References

• Training data poisoning → training-data-poisoning/SKILL.md
• Model inversion and memorization → model-inversion-membership-inference/SKILL.md
• Inference API abuse prevention → inference-api-abuse-prevention/SKILL.md
• Severity wording → severity-and-reporting.md