best-practices-prompt

# Prompt Best Practices (Project Skill) This skill is a curated set of rules for writing LLM prompts in this repo. Every prompt — system prompts, skill prompts, extraction templates, generation instructions — MUST pass these rules before going to `/review-prompt`. **The core problem:** Agents write vague, hand-wavy prompts that sound reasonable but produce garbage output. "Analyze the document and extract relevant information" is not a prompt — it is a wish. This skill exists to kill that pattern. ## When to Apply Use this skill whenever you: - write a new system prompt or skill prompt template - modify an existing prompt in a `.txt`, `.md`, or inline Python string - build prompts for `/scillm`, `/code-runner`, `/prompt-lab`, or any LLM call - prepare a prompt for `/review-prompt` review ## Categories (priority order) 1. Clarity (CRITICAL): `clarity-` — every sentence must have one unambiguous meaning 2. Specificity (CRITICAL): `specificity-` — concrete nouns, verbs, quantities, formats 3. Output Specification (HIGH): `output-` — exact schema, format, and constraints 4. Grounding (HIGH): `grounding-` — reference real data, files, schemas, examples 5. Structure (MEDIUM): `structure-` — logical ordering, sections, progressive disclosure 6. Testability (MEDIUM): `testability-` — how do you know the output is correct? 7. Efficiency (LOW): `efficiency-` — token budget, context window management ## Compliance Prompt Rules (STOP — read these first if writing SPARTA/NIST/CWE prompts) These rules apply to ANY prompt that operates on cross-framework compliance data (evidence cases, crosswalk chains, control glossaries, QRA results). Violations cause 0% accuracy. | Rule | What it prevents | Link | |------|-----------------|------| | `grounding-match-pipeline-schema` | Prompt invents its own schema instead of using the pipeline's actual output fields. Result: model can't parse the payload. | [Rule 15](#rule-15) | | `grounding-vocabulary-control` | Prompt says "different framework" without listing valid frameworks or telling the model how to identify them. Result: model guesses wrong. | [Rule 6](#rule-6) | | `grounding-cite-source-fields` | Prompt says "use the glossary" without naming `glossary[].framework`, `glossary[].description`. Result: model uses wrong fields. | [Rule 10](#rule-10) | **Before writing a compliance prompt:** 1. Find the function that produces the payload (e.g., `_build_evidence_case()` in `runner.py`) 2. Read its output — field names, types, nesting, framework identifiers 3. Use those exact field paths in the prompt 4. Include the framework closed vocabulary: SPARTA, CWE, NIST, CAPEC, ATT&CK, D3FEND ## Prompt Type Routing Load specialized rulesets based on prompt type: | Writing a... | Load this reference | Key rules | |--------------|---------------------|-----------| | **Extraction prompt** (QRA, entity, control extraction) | [`references/extraction.md`](references/extraction.md) | 24-36: source admissibility, distinctness, modality preservation | | **Inference/filter prompt** (classification, ranking, JSON output) | [`references/inference-time.md`](references/inference-time.md) | 16-23: format anchor, anti-MCQ, gates vs ranking, Pydantic patterns | | **Any prompt** | Core rules below | 0-15: no weasel words, concrete nouns, schema, examples | | **Debugging failures** | [`references/incidents.md`](references/incidents.md) | Pattern library of past prompt failures | ## Rule S: Models Never Return Prose (NON-NEGOTIABLE, operator 2026-09-24) ### Rule: `output-json-fields-only` Every model output is a JSON object whose fields are fixed by a Pydantic model. No model ever returns a free-prose answer, narration, summary, or explanation for a consumer to display or concatenate. Required pipeline: 1. **Model → JSON fields.** The prompt specifies the exact schema (Rules 2, 3, 12, 13) with hard quantities per field (e.g. `direct_answer`: one sentence, max 20 words; `steps[].point`: one sentence, max 15 words). 2. **Pydantic validation.** The consumer validates with a strict model (`extra="forbid"`, word/sentence-count validators, closed enums). Invalid output is rejected, never repaired into prose or truncated. 3. **`$jev` judgment.** A bounded Jev choice question scores the validated fields against the supplied evidence (answers-the-question and grounded-in-source distributions). `rejected`/`uncertain`/`unavailable` stay distinct and block acceptance; they are never coerced to success. 4. **Deterministic rendering.** Only code turns fields into display or speech (headline, source chip, numbered steps). Code never glues model text into paragraphs. Observed 2026-09-24: explain-project's `ExplainerStep.spoken: str` (max 2000 chars) let the authoring model write narration; the cockpit concatenated every step into an unscannable 110-word paragraph with no source file. ## Quick Reference (Core Rules 0-15) | Rule | Code | One-liner | |------|------|-----------| | 0 | `structure-rationale-header` | Every prompt starts with Purpose/Consumer/Why header | | 1 | `clarity-no-weasel-words` | No "relevant", "appropriate", "ensure", "comprehensive" | | 2 | `specificity-name-the-format` | Exact JSON schema, not "return JSON" | | 3 | `output-show-one-example` | Full input→output example, not fragments | | 4 | `clarity-imperative-voice` | "Extract X" not "you should try to extract X" | | 5 | `structure-task-before-context` | State task, THEN provide context | | 6 | `grounding-vocabulary-control` | List ALL valid categories explicitly | | 7 | `output-rejection-criteria` | Tell model what makes output WRONG | | 8 | `specificity-concrete-nouns` | "control ID, title, page" not "the information" | | 9 | `specificity-quantity-not-quality` | "2-3 sentences" not "brief summary" | | 10 | `grounding-cite-source-fields` | Name exact field paths: `doc.lineage.entity_ids` | | 11 | `structure-one-task` | One prompt = one task, chain for complex work | | 12 | `structure-schema-last` | Output schema at END, right before generation | | 13 | `output-nothing-but` | "Return ONLY this JSON, no other text" | | 14 | `testability-deterministic-check` | Define how to verify output is correct | | 15 | `grounding-match-pipeline-schema` | Prompt schema must match pipeline payload (compliance) | | 16 | `structure-review-payload` | Every prompt dir needs `review/` with complete payload for Web LLM audit | --- ## Rule 0: Every Prompt Must Have a Rationale Header (NON-NEGOTIABLE) ### Rule: `structure-rationale-header` Every prompt file MUST begin with a rationale block that answers three questions before the prompt itself starts. This block is for humans reviewing the prompt — it is NOT sent to the LLM. ``` # RATIONALE (not sent to LLM) # Purpose: What does this prompt produce? (e.g., "Extracts CWE→SPARTA crosswalk chains from evidence cases") # Consumer: What system/skill receives the output? (e.g., "SPARTA Explorer NLG pipeline, step 4") # Why this matters: What breaks if this prompt is wrong? (e.g., "Users get hallucinated attack paths") ``` ### Why this is Rule 0 Without rationale, prompts become orphans. Six months later, nobody knows: - **What** the prompt is for — is it extraction? classification? generation? - **Who** consumes the output — a Pydantic model? a human? another LLM? - **Why** it exists — what user-facing behavior depends on it? This is how "word salad" prompts survive: nobody can tell if the prompt is wrong because nobody knows what right looks like. The rationale makes the prompt auditable — a reviewer can check "does this prompt actually produce what the rationale says it should?" ### WRONG: ``` You are a cybersecurity analyst producing actionable insights from cross-framework evidence. [... 50 lines of instructions ...] ``` ### RIGHT: ``` # RATIONALE (not sent to LLM) # Purpose: Generate structured cybersecurity analysis from SPARTA evidence cases # Consumer: SPARTA Explorer API → /api/query response → "answer" field # Why this matters: This is the user-facing answer. Bad output = user sees nonsense. # Input: EVIDENCE_CASE JSON with glossary, crosswalk_chains, prior_qra_evidence # Output: JSON with decision, answer, citations. Validated by EvidenceCaseResponse Pydantic model. # Last reviewed: 2026-04-09 by Graham You are a NIST/SPARTA cross-framework evidence analyst. Task: Given an EVIDENCE_CASE, produce a structured cybersecurity analysis... ``` ### What belongs in the rationale: - **Purpose** — one sentence: what does this prompt do - **Consumer** — what code/system receives the output (skill name, API endpoint, Pydantic model) - **Why this matters** — what breaks if the output is wrong - **Input** — what data the prompt receives (field names, not vibes) - **Output** — schema name or format spec - **Last reviewed** — date and person, so staleness is visible ### How agents should handle the rationale: When calling `run.sh review`, the rationale block (lines starting with `#`) is parsed separately and shown in the review page header — not mixed into the prompt body. When the prompt is sent to the LLM, strip lines starting with `# RATIONALE` through the first non-comment line. --- ## Rule 1: No Weasel Words (NON-NEGOTIABLE) ### Rule: `clarity-no-weasel-words` Weasel words are adjectives and adverbs that feel meaningful but carry zero information for an LLM. They are the #1 source of vague prompts. If a human reader could reasonably ask "what do you mean by that?", it is a weasel word. **Banned words and what to replace them with:** | Weasel Word | Why It Fails | Replace With | |-------------|-------------|-------------| | "relevant" | Relevant to what? | Name the specific fields or criteria | | "appropriate" | By whose standard? | State the standard explicitly | | "comprehensive" | How comprehensive? | "All X" or "at least N" | | "thorough" | Unmeasurable | Enumerate what must be checked | | "important" | Important how? | State the consequence of missing it | | "ensure" | Not an action verb | "Check that X. If not, do Y." | | "consider" | Does not commit to action | "If X, then Y. Otherwise Z." | | "properly" | Properly by what definition? | State the exact criteria | | "meaningful" | Subjective | Define the threshold | | "high-quality" | Unmeasurable | List the quality criteria explicitly | | "as needed" | Who decides? | State the condition: "if X, then Y" | | "various" | How many? Which ones? | List them or say "all" | | "leverage" | Corporate jargon for "use" | "use" | | "utilize" | Same | "use" | ### WRONG: ``` Analyze the document and extract relevant information. Ensure comprehensive coverage of important topics. Provide appropriate responses as needed. ``` ### RIGHT: ``` Extract every person name, organization, and date from the document. Return them as a JSON array of objects: {"entity": "...", "type": "person|org|date", "page": N}. If no entities are found on a page, omit that page from the output. ``` ### Why this matters On 2026-03-22, a QRA extraction prompt using "extract relevant security controls" produced 40% hallucinated controls because "relevant" gave the LLM permission to invent plausible-sounding controls. Replacing it with "extract only controls whose ID appears verbatim in the source text" dropped hallucinations to 2%. --- ## Rule 2: Name the Output Format Exactly (NON-NEGOTIABLE) ### Rule: `specificity-name-the-format` Every prompt that expects structured output MUST specify the exact format. Not "return JSON" — return the exact JSON schema with field names, types, and constraints. ### WRONG: ``` Return the results as JSON. ``` ``` Output a structured response with the key findings. ``` ### RIGHT: ``` Return a JSON object with this exact schema: { "controls": [ { "id": "string — e.g. 'AC-2', 'SI-4'. Must match regex ^[A-Z]{2}-\\d+$", "title": "string — the control title from the source, verbatim", "source_page": "integer — 1-indexed page number where this control appears", "confidence": "float 0.0-1.0 — 1.0 if quoted verbatim, 0.7 if paraphrased" } ], "document_title": "string — from the PDF metadata or first heading", "total_pages_scanned": "integer" } Do not add fields not listed above. Do not nest objects deeper than shown. If a field cannot be determined, use null — never omit the key. ``` ### Why this matters "Return JSON" gives the LLM freedom to invent any schema. Every downstream consumer then needs to handle N possible shapes. Specifying the schema upfront means the output is parseable by a fixed Pydantic model — no guessing, no `get()` with fallbacks, no "try these 3 field names". --- ## Rule 3: Show One Complete Input/Output Example (NON-NEGOTIABLE) ### Rule: `output-show-one-example` Every prompt that produces structured output MUST include at least one complete input/output example. Not a fragment — the full input and the full expected output. ### WRONG: ``` Extract entities from the text. For example, you might find names like "John Smith". ``` ### RIGHT: ``` ## Example Input text: "On March 15, 2026, Dr. Sarah Chen from NIST published SP 800-171 Rev 3." Expected output: { "entities": [ {"text": "March 15, 2026", "type": "date", "start_char": 3, "end_char": 17}, {"text": "Dr. Sarah Chen", "type": "person", "start_char": 19, "end_char": 33}, {"text": "NIST", "type": "organization", "start_char": 39, "end_char": 43}, {"text": "SP 800-171 Rev 3", "type": "document_id", "start_char": 54, "end_char": 70} ] } Note: start_char and end_char are 0-indexed byte offsets into the input string. ``` ### Why this matters Examples are worth 100 lines of instruction. The LLM learns the output shape, the level of detail expected, the naming conventions, and the edge cases from a single concrete example. Without an example, each instruction sentence is another opportunity for misinterpretation. --- ## Rule 4: Imperative Voice, Not Descriptive (HIGH) ### Rule: `clarity-imperative-voice` Prompts are instructions, not descriptions. Use imperative verbs ("extract", "return", "list", "check") not descriptive language ("you should", "it would be good to", "the system is expected to"). ### WRONG: ``` You are a helpful assistant that analyzes documents. You should try to find security controls and it would be good to return them in a structured format. The system is expected to handle edge cases appropriately. ``` ### RIGHT: ``` You are a NIST SP 800-53 control extractor. Task: Extract every security control reference from the input document. Steps: