create-qras

# /create-qras Generate QRA pairs from any source: controls, documents, or raw text. ## Mandatory Preflight Before Any QRA Run Before running `review`, `manifest`, `generate`, or modifying this skill, the agent must read this entire `SKILL.md` from disk and print a read receipt. The receipt must include: - absolute `SKILL.md` path - line count - SHA-256 hash - selected mode: `native`, `relationship`, `standalone`, or `auto` - execution scope: `smoke`, `canary`, `corpus_repair`, or `skill_edit` - backend contract for the chosen scope - execution gate for the chosen scope - JSON repair path - fail-fast rule Required backend contract for large/corpus repair: - use `/v1/scillm/batch/completions` - use `model_pool: "qra-deepseek-pool"` by default - use stable `batch_id` - use stable item ids - use `response_format: {"type": "json_object"}` - do not hand-pick provider SDKs or call Claude/Codex OAuth for high-volume QRA generation Required execution gate for corpus repair: 1. `./run.sh review <manifest>` 2. inspect `BLOCKED` / `CANARY_ONLY` / `FULL_RUN_OK` 3. `./run.sh manifest <manifest> --limit <N> --dry-run` 4. small canary write 5. only then a larger reviewed batch Required JSON repair path: 1. project-local `.agents/skills/json_utils.py` 2. shared `skills/common/json_utils.py` fallback If `.agents/skills/json_utils.py` lacks real JSON extraction/repair for LLM output, stop and fix it before running QRA generation. Required fail-fast rule: - unrecovered item errors fail the manifest run - partial progress is not success - a run with skipped/error jobs must not be reported green If the read receipt is missing, the run is invalid even if it writes QRAs. ## Quick Decision: Which Mode Do I Use? | I want to answer... | Use mode | Example | |---------------------|----------|---------| | "What is T1595 Active Scanning?" | `--mode native` | Framework definition from MITRE ATT&CK | | "How does CWE-287 enable bypass of SPARTA IA-0001?" | `--mode relationship` | Cross-framework mapping with evidence chains | | "What does this PDF say about satellite security?" | `--mode standalone` | Knowledge extraction from documents | ## Modes (IMPORTANT - read this) ### native - Framework Definitions **Question type:** "What is X according to [framework]?" **Use when:** You need authoritative definitions from framework source documentation (ATT&CK, CWE, NIST, CAPEC, D3FEND). **Output category:** `attack_native`, `cwe_native`, `nist_native`, etc. ```bash # Generate native QRAs for ATT&CK technique ./run.sh generate --control T1595 --mode native # Batch generate for all ATT&CK Enterprise techniques ./run.sh generate --framework ATT_CK_Enterprise --mode native --limit 100 ``` **What it does:** 1. Loads control document from `sparta_controls` 2. Enriches with URL content from `sparta_url_knowledge` (for thin frameworks) 3. Uses v6 prompt with source admissibility rules, modality preservation 4. Generates 1-6 QRAs per control covering: definition, detection, mitigation, scope, risk **Category field:** `attack_native`, `cwe_native`, `nist_native`, `capec_native`, `d3fend_native`, `sparta_native` ### relationship - Cross-Framework Mappings **Question type:** "How does X relate to SPARTA Y?" **Use when:** You need to explain how a weakness/attack maps to SPARTA countermeasures via crosswalk chains. **Output category:** `sparta_context` (implicitly, via relationship) ```bash # Generate relationship QRA for CWE→SPARTA ./run.sh generate --control CWE-79 --mode relationship # Explicit source→target ./run.sh generate --source CWE-287 --target IA-0001 ``` **What it does:** 1. Finds SPARTA targets via `sparta_relationships` edges 2. Calls `/create-evidence-case` for crosswalk chains 3. Generates QRA explaining the relationship with grounded evidence **Requires:** Source control must have edges to SPARTA in `sparta_relationships`. ### standalone - Document Extraction **Question type:** "What does this document say about X?" **Use when:** You need to extract Q&A pairs from URL knowledge documents, PDFs, or fetched web content. ```bash # Generate from specific document ./run.sh generate --doc url_knowledge_12345 --mode standalone # Batch from collection ./run.sh generate --collection sparta_url_knowledge --mode standalone --limit 50 ``` **What it does:** 1. Loads document content 2. Extracts cybersecurity-relevant Q&A pairs 3. Stores with `qra_type: standalone` ### auto - Detect from Input (default) When `--mode auto` (default), mode is detected from input: | Input | Detected Mode | Why | |-------|--------------|-----| | `--control CWE-79` | relationship | CWE has crosswalk chains to SPARTA | | `--control CAPEC-115` | relationship | CAPEC has crosswalk chains to SPARTA | | `--control T1595` | native | ATT&CK - extract definitions | | `--control AC-17` | native | NIST - extract definitions | | `--control SV-AC-2` | native | SPARTA - extract definitions | | `--doc doc123` | standalone | Document extraction | ## Usage Examples ```bash # Native: ATT&CK technique definition ./run.sh generate --control T1595 --mode native # Native: Batch all ATT&CK Enterprise ./run.sh generate --framework ATT_CK_Enterprise --mode native --limit 500 # Relationship: CWE→SPARTA with evidence chains ./run.sh generate --control CWE-287 --mode relationship # Standalone: From URL knowledge ./run.sh generate --doc url_knowledge_xyz --mode standalone # Dry run: Preview what would be generated ./run.sh generate --control T1595 --mode native --dry-run # Dump prompts for human review (no LLM call) ./run.sh generate --control T1595 --mode native --dump-prompts ./review/ ``` ## Output Schema All QRAs share a common schema, with mode-specific fields: ```json { "_key": "qra_native_t1595_p1_abc123", "qra_id": "qra_native_t1595_p1_abc123", "run_id": "skill_create_qras_native_1712836800", "question": "What is T1595 Active Scanning according to MITRE ATT&CK?", "reasoning": "Scanning is classed as reconnaissance rather than access because it probes an exposed surface without exercising a vulnerability, so it precedes and informs later technique selection.", "answer": "T1595 Active Scanning is a reconnaissance technique...", "evidence_quotes": [ {"quote": "Adversaries may execute active reconnaissance scans...", "relevance": "Primary definition"} ], "qra_type": "native", "category": "attack_native", "source_framework": "ATT_CK_Enterprise", "source_control_id": "T1595", "sparta_linked": false, "pair_type": "threat_description", "confidence": "high", "actionable_for": "training", "prompt_version": "control_to_qra_v6", "generator": "skill:create-qras:native", "created_at": 1712836800, "verdict": "SATISFIED" } ``` ### Mode-Specific Fields | Field | native | relationship | standalone | |-------|--------|--------------|------------| | `qra_type` | "native" | "relationship" | "standalone" | | `category` | "attack_native", etc. | (sparta_context) | (standalone) | | `source_framework` | ATT&CK, CWE, etc. | CWE, CAPEC | - | | `target_framework` | - | SPARTA | - | | `crosswalk_chain` | - | ["CWE-79", "T1059", "SV-AC-2"] | - | | `source_doc` | - | - | doc key | | `sparta_linked` | false | true | false | ## Quality Gates Every generated QRA is validated: | Gate | Criteria | Required | |------|----------|----------| | `has_question` | Question field exists | Yes | | `has_reasoning` | Reasoning present and non-empty | Yes | | `reasoning_not_source` | Reasoning is not a verbatim copy of the source description | Yes | | `reasoning_unique` | Reasoning is not byte-identical to another QRA's reasoning | Yes | | `has_answer` | Answer > 5 words | Yes | | `has_evidence` | At least 1 evidence quote | Yes | | `grounding_verified` | Quotes appear in source | Score >= 0.5 | QRAs failing gates get `verdict: NEEDS_REVIEW` and should not be used. `has_reasoning` used to require only "> 10 words". Word count is not a proxy for reasoning: a fluent provenance sentence clears any length floor while explaining nothing. On 2026-07-30 that gate passed 2,579 QRAs sharing the single string "the weaknesses field explicitly provides the cwe id." Reasoning prompts now ask for subject-matter reasoning with no length cap, and the two gates above test the properties that actually matter — that the reasoning is not the source text, and not another QRA's. ## LLM Backend Uses `/scillm` proxy. For large QRA/default DeepSeek work, `create-qras` MUST use `POST /v1/scillm/batch/completions` with `model_pool: "qra-deepseek-pool"` by default, stable `batch_id`, stable item ids, and `response_format: {"type": "json_object"}`. The skill must not hand-pick provider SDKs or run high-volume QRA calls through Claude/Codex OAuth. If a pool lane is degraded during a live corpus repair, the operator may set `SCILLM_QRA_MODEL_POOL` to a scillm-defined QRA pool such as `qra-deepseek-opencode-pool`. This is still server-side scillm pooling; it is not permission to bypass scillm or call provider SDKs directly. Single/control smoke calls may use `/v1/chat/completions` with a family-specific profile such as `chutes-deepseek` or `oc-deepseek`; avoid `model: "text"` because broad cross-family fallback can change prompt/response behavior. Batch repair and corpus coverage work uses the server-side pool. | Priority | Model | Timeout | Notes | |----------|-------|---------|-------| | 1 | DeepSeek-V3.2-TEE | 300s | Primary | | 2 | DeepSeek-V3.1-TEE | 300s | First fallback | | 3 | DeepSeek-R1-0528-TEE | 300s | Reasoning model | | 4 | Kimi-K2.5-TEE | 180s | Fast alternative | | 5 | Qwen3-235B-A22B-Thinking | 180s | 100% grounding | | 6 | Qwen3.5-397B-A17B-TEE | 300s | Last resort | ### How Model Pools Work For QRA generation, scillm manages explicit model-pool lanes, not broad cross-family fallback: 1. **Request routing:** Skill submits QRA batch items to `qra-deepseek-pool`; scillm routes each item to configured DeepSeek-family lanes. 2. **Lane consistency:** The pool must stay inside the intended model family unless the operator explicitly selects another scillm-defined pool. 3. **Provider/catalog validation:** Before large corpus repair, verify `/v1/scillm/model-pools/qra-deepseek-pool/status` and the Chutes catalog show the configured DeepSeek lane is currently callable. 4. **Failure handling:** Retry transient lane failures, but do not silently switch QRA generation to an unrelated model family. **Pool lane configuration is defined in scillm, not project code.** The skill declares the pool and structured-output requirement; scillm selects the actual serving model/lane and records provider/model metadata. **Resource allocation:** Models are ordered by: - **Availability** — TEE variants have dedicated capacity, non-TEE share pools - **Cost** — DeepSeek models are cheapest, Qwen3.5-397B is most expensive - **Latency** — Smaller models (Kimi-K2.5) respond faster for simple prompts The proxy tracks real-time concurrency via `/v1/scillm/concurrency` and adjusts effective limits when 429s occur (adaptive backoff). ### Batching (server-side model pool) Server-side batch processing for maximum throughput: 1. Build a bounded `chunk_size=8` group of QRA work items. 2. Submit the whole chunk as one `/v1/scillm/batch/completions/stream` request with `model_pool` from `SCILLM_QRA_MODEL_POOL` or default `qra-deepseek-pool`, and multiple `items`. 3. Let scillm assign those items across the configured model-pool lanes. 4. Parse JSON through the project-local SPARTA helper (`.agents/skills/json_utils.py`) when available, falling back to the shared skill JSON repair helper (`skills/common/json_utils.py`) before treating malformed output as an error. 5. Per-result: call `/create-evidence-case` → enrich QRA with `evidence_case` field. 6. Store immediately via `store_callback` (crash-safe). **Key requirement:** do not send high-volume corpus repair as one-item `/v1/scillm/batch/completions` requests. The scillm pool assigns lanes by item index inside a batch; one-item calls repeatedly use index `0` and can starve secondary lanes such as OpenCode Go. - Stable `batch_id` + `item_id` through the server-side pool for durable resume. - `response_format: {"type": "json_object"}` so scillm JSON guard/repair is active before local parsing. - Dynamic pool health via `/v1/scillm/model-pools/<pool>/status`. - Streaming heartbeats identify in-flight item ids, lanes, providers, models, and elapsed seconds while long-running provider calls are still active. - Transient scillm/proxy failures (`429`, `502`, `503`, `504`, timeout) retry the chunk before the batch is marked failed. - Any unrecovered item error must fail the manifest run fast. Do not continue a repair batch after failed jobs and then treat partial progress as success. Preferred corpus-repair shape: submit each `chunk_size=8` chunk as one `/v1/scillm/batch/completions/stream` request with multiple `items`, then process `item_completed`, `item_failed`, and `item_replayed` events with the same crash-safe per-item storage semantics. Acceptable debugging fallback: one item per `/v1/scillm/batch/completions` request, but only for canaries, incident recovery, or while preserving crash-safe per-item storage during a code fix. Do not describe this fallback as the preferred batch shape. Corpus repair manifests should run in reviewed batches of 100 controls unless a