testing-for-system-prompt-leakage

name	testing-for-system-prompt-leakage
description	Extracts LLM system prompts using direct requests, jailbreak/instruction-override framing, translation/encoding tricks, and few-shot replay, combining manual payloads with automated garak and Promptfoo scanners to surface embedded secrets, routing logic, and policy leakage (OWASP LLM07:2025). Use during LLM application red-team engagements or when validating that no credentials or authorization logic live in the system prompt.
domain	cybersecurity
subdomain	ai-security
tags	["ai-security","system-prompt-leakage","prompt-extraction","llm-red-team","owasp-llm07","garak","promptfoo","data-leakage"]
version	1.0
author	mahipal
license	Apache-2.0
nist_ai_rmf	["MEASURE-2.7"]
atlas_techniques	["AML.T0057"]

Testing for System Prompt Leakage

Authorized use only: The extraction payloads below are for assessing LLM applications you own or have written authorization to test. Extracting prompts, secrets, or routing logic from systems you are not authorized to test may be unlawful.

Overview

A system prompt (a.k.a. developer message, preamble, or instructions) steers an LLM application's behavior. OWASP LLM07:2025 System Prompt Leakage addresses the risk that these prompts contain sensitive material that was never meant to be exposed — API keys, database connection strings, internal role/permission logic, model-routing rules, content policies, and tool definitions. Two principles frame this skill:

1. The system prompt must never be treated as a secret or used as a security control. If leaking it breaks your security model, the security model is wrong. The real findings during a leakage test are the secrets and logic embedded in the prompt that should have been enforced server-side.
2. System prompts are extractable. Through direct requests, instruction-override (jailbreak) framing, translation/encoding tricks, completion attacks, and few-shot replay, adversaries can reliably recover preambles.

This maps to MITRE ATLAS AML.T0057 — LLM Data Leakage: triggering unintentional disclosure (here, of the system prompt and embedded data) through crafted queries. Testing combines manual payloads with automated scanners — garak (NVIDIA's LLM vulnerability scanner) and Promptfoo (red-team eval) provide repeatable extraction probes.

When to Use

• During an LLM application penetration test or red-team engagement (OWASP LLM07 coverage).
• When validating that no secrets, credentials, or authorization logic live in the system prompt.
• When verifying that guardrails block prompt-extraction attempts.
• When building a regression suite that fails the build if a new prompt leaks.
• When assessing multi-agent or RAG apps where the preamble defines tool routing.

Prerequisites

• Authorization and scope covering the target LLM application/endpoint.
• Python 3.10+ for garak; Node.js 18+ for Promptfoo.
• API key or endpoint for the target model under test.

# garak — NVIDIA LLM vulnerability scanner
python -m pip install garak

# Promptfoo — red-team / eval framework
npm install -g promptfoo
# or run without install:  npx promptfoo@latest


python -m pip install openai