Run any Skill in Manus with one click

eval-threat-model

Stars16

Forks2

UpdatedJune 11, 2026 at 21:04

Developer/test skill that grades the SEMANTIC quality of a produced threat model (plausibility, severity proportionality, STRIDE coverage, mitigation actionability, missed surfaces) via a find→adversarial-verify loop over a frozen run. Complements the structural pytest/qa_checks suite; does NOT re-run the pipeline.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

matthiasrohr

matthiasrohr/appsec-advisor

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Software DevelopersComputer and Mathematical Occupations·SOC 15-1252

SKILL.md

readonly

name	eval-threat-model
description	Developer/test skill that grades the SEMANTIC quality of a produced threat model (plausibility, severity proportionality, STRIDE coverage, mitigation actionability, missed surfaces) via a find→adversarial-verify loop over a frozen run. Complements the structural pytest/qa_checks suite; does NOT re-run the pipeline.

eval-threat-model

Evaluate the quality of an already-produced threat model, not its structure. The pytest suite and qa_checks.py already prove the output is schema-valid, renders, and is cross-reference-clean. They cannot tell you whether the threats are plausible, the severities proportional, the STRIDE coverage complete, or the mitigations actionable. This skill does, by evaluating a frozen run (default: the bundled fixture) — no expensive pipeline re-run.

It runs a find→verify loop: per rubric dimension, one judge surfaces candidate defects, then a second, independent judge tries to refute them (refute-by-default). Only positively-grounded defects survive into EVAL-REVIEW.md. A deterministic Python harness (scripts/eval_threat_model.py) does the loading, signal computation, and scoring; the agents do only the irreducibly-semantic judging.

Inputs

Parse from the user's invocation; all optional:

RUN_DIR — dir holding threat-model.yaml. Default: $CLAUDE_PLUGIN_ROOT/tests/fixtures/e2e/frozen-run. For a real assessment, point it at the run's OUTPUT_DIR (dev-team default docs/security/, else your --output path), or tests/fixtures/e2e/_last-run for an E2E run. --keep-runtime-files is NOT required: the three inputs this skill reads — threat-model.yaml, .threats-merged.json (cwe/source/evidence enrichment), .recon-summary.md (recon context) — are all in runtime_cleanup.py's NEVER-delete set, so a completed run leaves them in place regardless. (The harness degrades gracefully if the two sidecars are absent — judges then work from threat-model.yaml prose alone.)
OUT — eval working/output dir. Default: $PWD/.eval-out.
REPO — optional target repo root. When set, judges may read the actual code to ground plausibility/missed-surface; when absent they judge from the brief.

Procedure

Step 1 — Prepare the brief (deterministic)

RUN_DIR="${RUN_DIR:-$CLAUDE_PLUGIN_ROOT/tests/fixtures/e2e/frozen-run}"
OUT="${OUT:-$PWD/.eval-out}"
REPO_ARGS=()
[ -n "$REPO" ] && REPO_ARGS+=(--repo "$REPO")

python3 "$CLAUDE_PLUGIN_ROOT/scripts/eval_threat_model.py" prepare \
  --run-dir "$RUN_DIR" --out "$OUT" "${REPO_ARGS[@]}"
PREP_EXIT=$?
[ "$PREP_EXIT" -ne 0 ] && { echo "eval-threat-model: prepare failed ($PREP_EXIT)"; exit "$PREP_EXIT"; }

The dimensions are fixed (5): stride_coverage, severity_proportionality, threat_plausibility, recommendation_actionability, missed_surface.

Step 2 — JUDGE fan-out (5 parallel sub-agents)

Dispatch five appsec-eval-judge sub-agents — one per dimension — via the Agent tool, in parallel. Do NOT inline the judging yourself. Inlining collapses the independent-verify guarantee and burns one serial context; the value of this skill is N independent judges. Each dispatch:

agent: appsec-eval-judge
MODE=JUDGE
DIMENSION=<one of the five>
BRIEF_PATH=<OUT>/brief.json
OUT_DIR=<OUT>
REPO_ROOT=<REPO if set, else omit>
MODEL_ID=sonnet

Each writes <OUT>/judge-<DIMENSION>.json. Wait for all five.

Step 3 — VERIFY fan-out (5 parallel sub-agents)

After all judge files exist, dispatch five fresh appsec-eval-judge sub-agents (independent instances = the adversarial check) — one per dimension, in parallel:

agent: appsec-eval-judge
MODE=VERIFY
DIMENSION=<one of the five>
BRIEF_PATH=<OUT>/brief.json
OUT_DIR=<OUT>
REPO_ROOT=<REPO if set, else omit>
MODEL_ID=sonnet

Each reads judge-<DIMENSION>.json and writes <OUT>/verify-<DIMENSION>.json (refute-by-default).

Step 4 — Aggregate + score (deterministic)

python3 "$CLAUDE_PLUGIN_ROOT/scripts/eval_threat_model.py" aggregate --out "$OUT"
AGG_EXIT=$?

aggregate keeps only real-verdict candidates plus the deterministic findings, scores them, and writes <OUT>/EVAL-REVIEW.md + <OUT>/eval-results.json. Exit code: 0 = no confirmed High/Critical defect, 1 = at least one (gate-able in CI), 2 = usage error.

Step 5 — Report

Relay only: the harness headline (confirmed counts + dropped-by-verify count), the EVAL-REVIEW.md path, and the aggregate exit code. Do not re-summarize every finding — the report is the source of truth. If AGG_EXIT == 1, surface the confirmed High/Critical titles so the user sees what to act on.

Notes

This is a test/dev skill — never auto-triggered, not part of create-threat-model. Run it manually after a pipeline change to catch quality regressions a frozen fixture would otherwise hide.
Cost is bounded: 10 sub-agents over a compact brief, no STRIDE/recon re-run.
The judges are skeptics; an empty EVAL-REVIEW.md ("no confirmed defects") is a real, valid result, not a failure.

More from this repository

same repository

create-threat-model

matthiasrohr/appsec-advisor

Perform a threat assessment of a repository and produce a threat-model.md. Supports --repo to analyze external repos and --output to set the output directory. Optionally also writes threat-model.yaml with --yaml flag.

2026-07-2516

diagnose-bundle

matthiasrohr/appsec-advisor

Maintainer/dev skill that triages an anonymised diagnostic bundle (appsec-diag-*.tgz, produced by scripts/diagnostic_bundle.py) a user sent after a pipeline failure. Runs the deterministic inspect, then cross-references the plugin source (scripts/, agents/phases/, AGENTS.md) and known-bug history to produce a grounded root-cause hypothesis. Does NOT re-run the pipeline and never needs the user's repo.

2026-07-2016

ask-threat-model

matthiasrohr/appsec-advisor

Answer ANY question about the threat model in this repo — read-only Q&A over the committed threat-model.yaml. The default surface for every natural-language query about the model, however simple: does one exist at all ("is there a threat model here?", "gibt es hier ein bedrohungsmodell?"), how it stands ("how bad is it?", "is it still current?"), anything about its contents ("what are the critical findings?", "does it cover SSRF / IDOR / auth?", "which findings touch the payment service?", "is there a fix for F-003?", "welche kritischen findings gibt es?", "deckt mein bedrohungsmodell XSS ab?"), and meta questions ("what does P1 mean?", "what is STRIDE here?"). Grounds every data answer in the model and cites F-ids; never analyzes code, re-scores, spawns agents, or writes files. Prefer this over show-threat-model for anything phrased as a question — show only prints a fixed summary block on explicit request. To act on findings use review-threat-model; to (re)generate use create-threat-model.

2026-07-1916

show-threat-model

matthiasrohr/appsec-advisor

Print the standard summary block of the current threat model — scan identity, findings by severity, the remediation backlog (P1/P2/P3) and mitigation coverage, the worst-case scenarios, control posture, and whether the model is still up to date. A DISPLAY command, not a question-answering surface: use it only for an explicit request to see that block ("show me the threat model summary", "zeig mir die threat-model-uebersicht", "print the overview"). Reuses the same change detection that decides if an incremental scan is needed. Does not analyze code, write files, or compose prose — it prints a rendered block verbatim and cannot answer anything the block does not already contain. Do NOT route a natural-language question here, not even "is there a threat model?" / "gibt es hier ein bedrohungsmodell?" — every question goes to ask-threat-model, which answers directly and can produce this block too.

2026-07-1816

review-threat-model

matthiasrohr/appsec-advisor

User-facing triage of an existing threat model — an overview-first console over an already-generated threat-model.yaml (backlog by priority, severity mix, worst-case scenarios), then one of three modes. "Just look around" browses findings read-only by severity, security aspect, requirement, or control posture. "Fix or accept findings now" applies the code fix for findings you pick — or accepts the risk instead — one at a time for review. "Build a remediation plan" decides mitigate / accept-risk per finding and emits a remediation-plan.md without code changes. Within a session you can also discuss a single finding by id (F-/T-/M-/W-), a read-only aside that changes nothing. Runs later and independently of create-threat-model; reads the model, never regenerates or re-scores it. Not an artifact-quality check (that is eval-threat-model). To only *ask* about the model without acting on it, use ask-threat-model.

2026-07-1816

update-threat-model

matthiasrohr/appsec-advisor

Convenience alias for create-threat-model in incremental mode — re-analyzes only the components that changed since the last run. Unlike the bare create-threat-model, it does NOT bootstrap — it aborts with a clear message when no threat model exists yet, so an "update" can never silently turn into a first full assessment. An explicit mode flag the user passes (--full, --rebuild, --rerender, --resume) is honored instead of the injected --incremental. Everything else (flags, --repo, --output, behavior, output) is identical to create-threat-model.

2026-07-1816

name	eval-threat-model
description	Developer/test skill that grades the SEMANTIC quality of a produced threat model (plausibility, severity proportionality, STRIDE coverage, mitigation actionability, missed surfaces) via a find→adversarial-verify loop over a frozen run. Complements the structural pytest/qa_checks suite; does NOT re-run the pipeline.

eval-threat-model

Inputs

Parse from the user's invocation; all optional:

RUN_DIR — dir holding threat-model.yaml. Default: $CLAUDE_PLUGIN_ROOT/tests/fixtures/e2e/frozen-run. For a real assessment, point it at the run's OUTPUT_DIR (dev-team default docs/security/, else your --output path), or tests/fixtures/e2e/_last-run for an E2E run. --keep-runtime-files is NOT required: the three inputs this skill reads — threat-model.yaml, .threats-merged.json (cwe/source/evidence enrichment), .recon-summary.md (recon context) — are all in runtime_cleanup.py's NEVER-delete set, so a completed run leaves them in place regardless. (The harness degrades gracefully if the two sidecars are absent — judges then work from threat-model.yaml prose alone.)
OUT — eval working/output dir. Default: $PWD/.eval-out.
REPO — optional target repo root. When set, judges may read the actual code to ground plausibility/missed-surface; when absent they judge from the brief.

Procedure

Step 1 — Prepare the brief (deterministic)

RUN_DIR="${RUN_DIR:-$CLAUDE_PLUGIN_ROOT/tests/fixtures/e2e/frozen-run}"
OUT="${OUT:-$PWD/.eval-out}"
REPO_ARGS=()
[ -n "$REPO" ] && REPO_ARGS+=(--repo "$REPO")

python3 "$CLAUDE_PLUGIN_ROOT/scripts/eval_threat_model.py" prepare \
  --run-dir "$RUN_DIR" --out "$OUT" "${REPO_ARGS[@]}"
PREP_EXIT=$?
[ "$PREP_EXIT" -ne 0 ] && { echo "eval-threat-model: prepare failed ($PREP_EXIT)"; exit "$PREP_EXIT"; }

The dimensions are fixed (5): stride_coverage, severity_proportionality, threat_plausibility, recommendation_actionability, missed_surface.

Step 2 — JUDGE fan-out (5 parallel sub-agents)

agent: appsec-eval-judge
MODE=JUDGE
DIMENSION=<one of the five>
BRIEF_PATH=<OUT>/brief.json
OUT_DIR=<OUT>
REPO_ROOT=<REPO if set, else omit>
MODEL_ID=sonnet

Each writes <OUT>/judge-<DIMENSION>.json. Wait for all five.

Step 3 — VERIFY fan-out (5 parallel sub-agents)

After all judge files exist, dispatch five fresh appsec-eval-judge sub-agents (independent instances = the adversarial check) — one per dimension, in parallel:

agent: appsec-eval-judge
MODE=VERIFY
DIMENSION=<one of the five>
BRIEF_PATH=<OUT>/brief.json
OUT_DIR=<OUT>
REPO_ROOT=<REPO if set, else omit>
MODEL_ID=sonnet

Each reads judge-<DIMENSION>.json and writes <OUT>/verify-<DIMENSION>.json (refute-by-default).

Step 4 — Aggregate + score (deterministic)

python3 "$CLAUDE_PLUGIN_ROOT/scripts/eval_threat_model.py" aggregate --out "$OUT"
AGG_EXIT=$?

Step 5 — Report

Notes

This is a test/dev skill — never auto-triggered, not part of create-threat-model. Run it manually after a pipeline change to catch quality regressions a frozen fixture would otherwise hide.
Cost is bounded: 10 sub-agents over a compact brief, no STRIDE/recon re-run.
The judges are skeptics; an empty EVAL-REVIEW.md ("no confirmed defects") is a real, valid result, not a failure.