llm-output-validation

name	llm-output-validation
description	Post-schema semantic validation checklist for structured LLM output. Use when writing code that calls an LLM and parses the response into a data model, reviewing code that consumes structured LLM output, or debugging cases where an LLM returned valid but wrong data.
license	MIT

LLM Output Validation

When an LLM produces structured data, schema validation (Pydantic, JSON Schema, etc.) only guarantees format correctness. It cannot catch semantically wrong but schema-valid output: hallucinated identifiers, out-of-scope selections, impossible values. Treat structured LLM output the same way you treat untrusted user input — validate meaning, not just shape.

When to Use

The boundary here is after the bytes parse: schema/Pydantic validation already passed, now validate meaning (hallucinated ids, out-of-scope picks, impossible values). Not prompt design or model choice — for the Anthropic API surface (params, tool-use, streaming) see the claude-api reference. Not generic input validation either; this is specifically for output you got from an LLM.

Validation Checklist

Apply these checks in code, immediately after model_validate() / schema parse. Raise on failure (loud, not a warning).

1. Input Round-Trip

Every identifier in the output must exist in the input. No invented IDs.

• If you passed bundle_id="001", the output must reference "001", not "002".
• If you passed a list of filenames, every filename in the response must appear in the original list.
• Implementation: set/dict membership check on every ID field.

2. Scope Subset

When the output selects from the input (e.g., "pick the top 3"), verify selection ⊆ input.

• If the input has items [A, B, C] and the output picks [A, D], D is hallucinated.
• Implementation: assert set(output.selected) <= set(input.candidates).

3. Value Bounds

Numeric fields must be within the declared range.

• Scores, ratings, confidence levels: check 0 <= v <= max.
• Counts: non-negative, bounded by input size.
• Implementation: simple range assertions or Pydantic Field(ge=0, le=10).

4. Reference Integrity

String references to entities (filenames, function names, URLs, paths) must resolve in the current context.

• A cited filename must exist on disk (or in the provided listing).
• A cited function must appear in the AST / grep output.
• Implementation: Path(ref).exists() or equivalent lookup.

5. Semantic Plausibility

Catch generic garbage the LLM emits when confused.

• Non-empty fields where content is required.
• No boilerplate filler ("As an AI language model…", "I'm sorry…").
• Field lengths proportional to expected content (a one-word "summary" of a 1000-line file is suspicious).
• Implementation: length checks, regex for known filler patterns.

Where to Put the Check

# Immediately after schema parse — before ANY downstream use.
result = ResponseModel.model_validate(raw)
validate_response_integrity(result, original_input)  # ← HERE
# Now safe to use `result`.

Raise ValueError (or a domain-specific error) on failure. Do not log a warning and continue — the whole point is to prevent downstream consumers from acting on garbage.

Case Study: HATS-066

An LLM in the (then-current) retrospective pipeline was asked to evaluate bundle_id="001". It returned a schema-valid result object with bundle_id="002" — a hallucinated identifier. Pydantic didn't notice because bundle_id was typed as str, not constrained to the input. The fix: a three-line integrity check after parse that verified result.bundle_id == expected_bundle_id. Trivial code, but it caught a class of bug that no amount of prompt engineering could reliably prevent.

Anti-Patterns

• Trusting Pydantic schema as a semantic guarantee — schema validates shape, not meaning.
• Logging a warning instead of raising — downstream code will still consume the bad data, just with a log line nobody reads.
• Deferring validation to a downstream consumer — by the time the consumer notices, the provenance is lost and the error is harder to debug.
• Adding constraints only to the prompt — prompts are suggestions, not contracts. Code is the only reliable enforcement.