llm-output-safety-reviewer

name

llm-output-safety-reviewer

description

Review how an application consumes LLM output for improper output handling (OWASP LLM05) — treat every model output as untrusted data and trace it to each sink: HTML/markdown rendering (XSS), SQL/NoSQL/shell/eval execution (injection, RCE), file paths and URLs (traversal, SSRF), tool/function arguments, and stored-then-re-consumed content (second-order). Verify context-correct encoding/escaping, sandboxing for executed generated code — incl. autonomous generate-and-run loops, sandbox escape, in-sandbox persistence, and natural-language-driven execution (agentic ASI05) — and validate-before-act discipline. Use when model output is rendered, executed, stored, or used to build a command, query, path, or request. Do NOT use for schema-shape validation of structured output (structured-output-validator), the injection that produced the output (prompt-injection-defender), factual correctness (ai-misinformation-guard), or tool-permission scope (agent-tool-safety-guard).

LLM Output Safety Reviewer

Purpose

Review whether an application handles model output safely (LLM05): the model output is untrusted data, and every place it flows — a rendered page, a database query, a shell command, a file path, a URL, a tool argument, or a store it will later be read from — is a potential injection or execution sink. The review traces output to each sink and verifies context-correct encoding/escaping, sandboxing for executed generated code, and validate-before-act discipline, producing severity-ranked findings each with the flow from model output to impact.

Use When

• Use when: model output is rendered as HTML/markdown, executed as SQL/shell/ code, written to files, used to build URLs or requests, passed as tool arguments, or stored and later re-consumed.
• Use when: reviewing an AI feature's output path for XSS, injection, RCE, SSRF, path traversal, or second-order (stored-output) issues.
• Use when: an agent generates code that the system then runs.
• Use when: an AUTONOMOUS agent loop generates and executes code with no human between generate and run (ASI05) — sandbox boundaries and escape paths, in-sandbox persistence between runs, package installs inside the sandbox, and which natural-language inputs can reach an execution path.
• Do NOT use when: the concern is the SHAPE/type of structured output — structured-output-validator (this skill is the injection/exec sink review; they compose).
• Do NOT use when: the concern is the injection that manipulated the model (prompt-injection-defender), factual accuracy (ai-misinformation-guard), or how broad the tools are (agent-tool-safety-guard).

Inputs to Inspect

1. The output sinks: every place the model's output is used — templates/JSX, query builders, command execution, file I/O, HTTP clients, tool-call dispatch, storage writes.
2. The rendering path: is output treated as text or as markup; is auto-escaping on; is dangerouslySetInnerHTML/v-html/innerHTML or a markdown renderer with raw-HTML enabled in play.
3. Execution paths: any eval, dynamic SQL, shell-out, code interpreter, or generated-code runner; the sandbox (or absence) around it. For agent loops (ASI05): what natural-language input can trigger execution, what persists in the sandbox between runs, and whether executed code can install packages or reach the network.
4. URL/path construction from output: SSRF (model-chosen URL fetched server-side), path traversal (model-chosen filename).
5. Storage-and-reuse: output persisted then later rendered/executed as if trusted (second-order injection).
6. Existing encoding/validation: escaping helpers, allowlists, content security policy, and where they are and aren't applied.

Workflow

1. Enumerate output sinks. List every consumer of model output and classify each: render, execute, store, or drive-an-action. No output- handling code to inspect → Stop Conditions.
2. Treat output as untrusted. For each sink, ask the same question you'd ask of raw user input: what happens if the output is adversarial? Model output is attacker-influenced whenever untrusted content was in context.
3. Review rendering sinks using references/output-sink-catalog.md: HTML/markdown must be context-correctly encoded or sanitized; raw-HTML rendering of model output is a finding unless sanitized with an allowlist; check CSP as defense in depth.
4. Review execution sinks. Model output that becomes SQL, shell, eval, or generated code that runs: require parameterization/allowlisting, and for generated-code execution require a real sandbox (isolated, no secrets, no network unless required, resource-limited). Unsandboxed execution of generated code is RCE-class. For AUTONOMOUS generate-and-run loops (ASI05): require per-run ephemeral sandboxes (no cross-run persistence a poisoned run can plant into), package installs treated as untrusted supply-chain events, an execution budget, and a map of every natural-language path that reaches execution — "user asks a question" → "agent writes and runs code" is an NL-to-RCE path to enumerate, not a feature to assume safe.
5. Review URL/path/request sinks. Server-side fetch of a model-chosen URL is SSRF — require an allowlist and block internal ranges/metadata endpoints. Model-chosen file paths need traversal-safe resolution.
6. Review tool-argument sinks. Output used as tool arguments must be validated before the side effect (compose structured-output-validator for shape, agent-tool-safety-guard for the permission boundary).
7. Review store-and-reuse. Output persisted and later rendered/executed is re-untrusted on read: encoding at write time is not enough if a different reader trusts it. Flag second-order paths.
8. Rank findings by flow. Each finding names the flow (model output → sink → impact) and severity gated on a concrete exploit; give the context-correct fix (escape here, parameterize there, sandbox, allowlist).

Output Format

LLM OUTPUT-HANDLING REVIEW — <feature>
Output sinks: <render | execute | store | action — each location>
Findings (severity-ranked):
  [SEV] <sink type> at <file:line>
    Flow: <model output → sink → impact (XSS/RCE/SSRF/injection/2nd-order)>
    Fix: <context-correct encoding | parameterize | sandbox | allowlist>
Sandbox posture (if code exec): <isolation, secrets, network, limits>
Second-order paths: <stored output re-consumed as trusted>
Defense-in-depth: <CSP, output length caps, content types>
Not reviewed: <areas + why>

Validation Checklist

• Every model-output sink enumerated and classified (render/execute/ store/action).
• Rendering sinks use context-correct encoding; raw-HTML rendering is sanitized with an allowlist or flagged.
• Execution sinks are parameterized/allowlisted; generated-code execution runs in a real sandbox or is flagged as RCE-class.
• Autonomous generate-and-run loops (if any) use per-run ephemeral sandboxes with no default secrets/network, and every natural-language path that reaches execution is mapped (ASI05).
• URL sinks checked for SSRF (allowlist, internal-range block); path sinks checked for traversal.
• Tool-argument sinks validate before the side effect (composed with structured-output-validator / agent-tool-safety-guard).
• Stored-then-reused output is treated as untrusted on read (second-order).
• Findings name the output→sink→impact flow; severity is exploit-gated.

Security Rules

• Model output is untrusted data, handled with the same discipline as raw user input — no sink gets to assume it's clean.
• Encoding is context-specific: HTML-escaping does not make output safe for a SQL string, a shell command, a URL, or a filename. Match the sink.
• Generated code that executes requires a sandbox; "we prompt it to write safe code" is not a control.
• Escaping at write time does not sanitize a later trusting read — second-order sinks re-validate.

Gotchas

• Markdown renderers often pass through raw HTML by default — a model emitting <img onerror=…> inside markdown becomes XSS unless raw HTML is disabled or sanitized.
• Streaming output tempts teams to render incrementally before any sanitization runs — the sanitize step must not be skipped for the stream.
• "It's just displayed, not executed" ignores that display IS execution for a browser — HTML/JS renders.
• SSRF via model output is easy to miss: the model helpfully returns a URL and the server fetches it to "follow the link" — straight into the metadata endpoint.
• The exploit often lands one hop away: output stored today, rendered in an admin console tomorrow by code that assumes internal data is safe.
• Don't conflate with shape validation: JSON that parses cleanly can still carry an XSS payload in a string field — shape-valid ≠ safe-to-render.
• Agent loops normalize execution (ASI05): when generate→run fires dozens of times an hour autonomously, one hijacked generation is RCE with nobody watching — and a sandbox that persists state between runs lets a poisoned run plant what the next run trusts (pip install into a shared venv is the classic). Ephemeral per-run sandboxes and NL-path mapping are the controls, not human review of each generation.

Stop Conditions

• No output-handling code (templates, query builders, exec paths) is available — stop; this skill reviews concrete sinks, not a description.
• The review finds generated code executing unsandboxed with access to secrets or the network — flag as blocking and route remediation through human-approval-boundary.
• The real issue is the model being manipulated (injection), output shape, or factual correctness — hand to the owning skill.
• A live exploit is evident (active XSS/RCE via output) — route to incident-response-runbook.

Supporting Files

• references/output-sink-catalog.md — per-sink review checklist (render/execute/URL/path/tool-arg/store), context-correct encoding rules, the generated-code sandbox rubric (including the ASI05 autonomous-loop and sandbox-escape extension), and second-order injection patterns.
• evals/evals.json — trigger + behavior cases.
• evals/trigger-evals.json — discrimination within the output & agency cluster and against security-pr-reviewer and structured-output-validator.