Run any Skill in Manus with one click

benchmark

Stars0

Forks0

UpdatedJune 10, 2026 at 10:46

Measure recall / precision / false-proof rate of the pipeline against a ground-truth manifest. Scores either the bundled planted-vulnerability corpus (regression) or a live run's findings.json against a manifest you supply. Deterministic — no agent, no network. Use to prove a change to the producers helps rather than hurts.

Installation

Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.

Run Skill in Manus

Source

allsmog

allsmog/kuzushi-security-plugin

View GitHub Repository View Creator Repositories

Download

Run Skill in Manus

Related occupationsSOC

Based on SOC occupation classification

Information Security AnalystsComputer and Mathematical Occupations·SOC 15-1212

SKILL.md

readonly

name	benchmark
description	Measure recall / precision / false-proof rate of the pipeline against a ground-truth manifest. Scores either the bundled planted-vulnerability corpus (regression) or a live run's findings.json against a manifest you supply. Deterministic — no agent, no network. Use to prove a change to the producers helps rather than hurts.
context	inline
user-invocable	true

Benchmark

You can't call bug-finding "world-class" — or catch a regression in it — without a number. /benchmark scores findings against ground truth and reports the three metrics that matter: recall (are we missing bugs?), precision (do we cry wolf?), and falseProofRate (did we prove a non-bug? — the soundness failure differential testing guards).

Run it

Bundled corpus (regression): node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" scores every case under bench/cases/ using its recorded findings.json. Add --case <name> for one case.
A live run: node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" --target "<repo>" --ground-truth "<manifest.json>" scores <repo>/.kuzushi/findings.json after you've run the pipeline.

Flags: --strict (an active finding matching no expectation counts as a false positive — only fair when the manifest is exhaustive), --line-tolerance N (default 5), --no-match-cwe (match on file+line only).

Ground-truth manifest

{ "expectations": [ { "id", "kind": "vuln" | "safe", "cwe", "filePath", "line" } ] }. A vuln is a real bug the tool should find; a safe is a decoy that looks like one and must not be flagged. A decoy that gets an active finding is a false positive; a decoy that gets a proven finding is a false proof. Author manifests from confirmed bugs (and their guarded siblings) so the corpus encodes both recall and precision pressure.

Reading the result

corpus aggregates across cases; cases[].perExpectation shows each hit/miss. A drop in recall means a producer started missing a bug class; a drop in precision or any falseProofs means it started over-promoting. Wire the corpus run into CI so either regresses loudly.

When NOT to use

To find bugs — /benchmark measures a run that already happened; it runs no analysis.
On a target with no ground-truth manifest — without annotated expectations there is nothing to score against; write the manifest first (or use the bundled corpus).
As a security verdict — a green benchmark means the corpus didn't regress, not that a given repo is bug-free.

Rationalizations to Reject

"Recall is 1.0 on the corpus, so the tool finds everything." → The corpus is a tiny planted set; it proves non-regression on known shapes, not coverage of the unknown. Grow the corpus when you confirm a new bug class, or the number lies by omission.
"Precision dipped but the new hits look plausible." → Plausible ≠ correct. A dip means a decoy got flagged; open it and confirm the guard before accepting the change.
"falseProofs went up but the bugs are real." → A false proof is a proven hit on a decoy — the harness fired on a non-bug. That's a soundness break regardless of intent; fix the harness/differential gate before shipping.

More from this repository

same repository

deep-hunt

allsmog/kuzushi-security-plugin

Interprocedural, hypothesis-driven hunt — finds the multi-file source→sink bugs that pattern-gating and same-file taint miss. Ranks trace anchors (entry points + dangerous sinks), then the deep-hunter agent walks each flow ACROSS files (forward/backward call-graph CLIs) over multiple rounds, confirming propagation and defeating guards, and promotes confirmed cross-file flows into findings.json with the path stored as evidenceGraph. Token-expensive, budget-bounded. Part of the HUNT phase (run via /sweep --deep, or ask for it); leads flow to /verify (panel).

2026-06-110

deep-scan

allsmog/kuzushi-security-plugin

Whole-file deep reader — finds bugs by READING risk-ranked files in full and reasoning from first principles, not by grepping for known patterns. Catches the long tail (project-specific wrappers, business logic, cross-function flaws) that pattern-gated producers structurally miss. Token-expensive; budget-bounded and risk-ranked. Promotes leads into .kuzushi/findings.json (source "deep-scan").

2026-06-110

taint-analysis

allsmog/kuzushi-security-plugin

IRIS-style source→sink taint hunt. Ranks a typed CWE catalog for the repo, then drives subagents to label dangerous sinks, label sources of user input, run Joern/CodeQL dataflow queries (or same-file linking) to connect them, and triage each flow as finding/candidate/rejected. Promotes verdicts into .kuzushi/findings.json. Benefits from a prebuilt CodeQL DB / Joern CPG but degrades gracefully.

2026-06-110

threat-hunt

allsmog/kuzushi-security-plugin

Adversarial per-threat review (Carlini doctrine). For each threat in .kuzushi/threat-model.json, state attacker capabilities, walk source→sink, attempt to bypass every guard, and assign a verdict with evidence. Promotes findings into .kuzushi/findings.json. Requires /threat-model first.

2026-06-110

partition

allsmog/kuzushi-security-plugin

Split the attack surface into parallel-discovery partitions so fan-out hunters explore different subsystems instead of converging on the same shallow bug. Deterministic grouping of /x-ray entry points by component → .kuzushi/partitions.json, which a hunt coordinator hands to one subagent per partition. Run /x-ray first.

2026-06-100

poc

allsmog/kuzushi-security-plugin

Empirical proof-of-concept for the PoC-ready findings. For each finding /verify marked confirmed-exploitable or inconclusive, synthesize a minimal harness that triggers the bug; a host script then runs it in a sandbox (Docker --network none when present, else a gated local run) and classifies the crash into a proof verdict. Attaches a poc block onto each finding. Requires /verify first.

2026-06-100

name	benchmark
description	Measure recall / precision / false-proof rate of the pipeline against a ground-truth manifest. Scores either the bundled planted-vulnerability corpus (regression) or a live run's findings.json against a manifest you supply. Deterministic — no agent, no network. Use to prove a change to the producers helps rather than hurts.
context	inline
user-invocable	true

Benchmark

Run it

Bundled corpus (regression): node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" scores every case under bench/cases/ using its recorded findings.json. Add --case <name> for one case.
A live run: node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" --target "<repo>" --ground-truth "<manifest.json>" scores <repo>/.kuzushi/findings.json after you've run the pipeline.

Ground-truth manifest

Reading the result

When NOT to use

To find bugs — /benchmark measures a run that already happened; it runs no analysis.
On a target with no ground-truth manifest — without annotated expectations there is nothing to score against; write the manifest first (or use the bundled corpus).
As a security verdict — a green benchmark means the corpus didn't regress, not that a given repo is bug-free.

Rationalizations to Reject

"Recall is 1.0 on the corpus, so the tool finds everything." → The corpus is a tiny planted set; it proves non-regression on known shapes, not coverage of the unknown. Grow the corpus when you confirm a new bug class, or the number lies by omission.
"Precision dipped but the new hits look plausible." → Plausible ≠ correct. A dip means a decoy got flagged; open it and confirm the guard before accepting the change.
"falseProofs went up but the bugs are real." → A false proof is a proven hit on a decoy — the harness fired on a non-bug. That's a soundness break regardless of intent; fix the harness/differential gate before shipping.