원클릭으로 Manus에서 모든 스킬 실행

benchmark

스타0

포크0

업데이트2026년 6월 10일 10:46

Measure recall / precision / false-proof rate of the pipeline against a ground-truth manifest. Scores either the bundled planted-vulnerability corpus (regression) or a live run's findings.json against a manifest you supply. Deterministic — no agent, no network. Use to prove a change to the producers helps rather than hurts.

설치

Codex 또는 Claude로 설치 이 Prompt를 복사해 Codex, Claude 또는 다른 어시스턴트에 붙여 넣으면 Skill 페이지를 검토하고 설치를 진행할 수 있습니다.

Manus에서 실행

출처

allsmog

allsmog/kuzushi-security-plugin

GitHub 저장소 열기 Creator 저장소 보기

다운로드

Manus에서 실행

Benchmark

You can't call bug-finding "world-class" — or catch a regression in it — without a number. /benchmark scores findings against ground truth and reports the three metrics that matter: recall (are we missing bugs?), precision (do we cry wolf?), and falseProofRate (did we prove a non-bug? — the soundness failure differential testing guards).

Run it

Bundled corpus (regression): node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" scores every case under bench/cases/ using its recorded findings.json. Add --case <name> for one case.
A live run: node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" --target "<repo>" --ground-truth "<manifest.json>" scores <repo>/.kuzushi/findings.json after you've run the pipeline.

Flags: --strict (an active finding matching no expectation counts as a false positive — only fair when the manifest is exhaustive), --line-tolerance N (default 5), --no-match-cwe (match on file+line only).

Ground-truth manifest

{ "expectations": [ { "id", "kind": "vuln" | "safe", "cwe", "filePath", "line" } ] }. A vuln is a real bug the tool should find; a safe is a decoy that looks like one and must not be flagged. A decoy that gets an active finding is a false positive; a decoy that gets a proven finding is a false proof. Author manifests from confirmed bugs (and their guarded siblings) so the corpus encodes both recall and precision pressure.

Reading the result

corpus aggregates across cases; cases[].perExpectation shows each hit/miss. A drop in recall means a producer started missing a bug class; a drop in precision or any falseProofs means it started over-promoting. Wire the corpus run into CI so either regresses loudly.

When NOT to use

To find bugs — /benchmark measures a run that already happened; it runs no analysis.
On a target with no ground-truth manifest — without annotated expectations there is nothing to score against; write the manifest first (or use the bundled corpus).
As a security verdict — a green benchmark means the corpus didn't regress, not that a given repo is bug-free.

Rationalizations to Reject

"Recall is 1.0 on the corpus, so the tool finds everything." → The corpus is a tiny planted set; it proves non-regression on known shapes, not coverage of the unknown. Grow the corpus when you confirm a new bug class, or the number lies by omission.
"Precision dipped but the new hits look plausible." → Plausible ≠ correct. A dip means a decoy got flagged; open it and confirm the guard before accepting the change.
"falseProofs went up but the bugs are real." → A false proof is a proven hit on a decoy — the harness fired on a non-bug. That's a soundness break regardless of intent; fix the harness/differential gate before shipping.

이 저장소의 다른 Skills

같은 저장소

deep-hunt

allsmog/kuzushi-security-plugin

Interprocedural, hypothesis-driven hunt — finds the multi-file source→sink bugs that pattern-gating and same-file taint miss. Ranks trace anchors (entry points + dangerous sinks), then the deep-hunter agent walks each flow ACROSS files (forward/backward call-graph CLIs) over multiple rounds, confirming propagation and defeating guards, and promotes confirmed cross-file flows into findings.json with the path stored as evidenceGraph. Token-expensive, budget-bounded. Part of the HUNT phase (run via /sweep --deep, or ask for it); leads flow to /verify (panel).

2026-06-110

deep-scan

allsmog/kuzushi-security-plugin

Whole-file deep reader — finds bugs by READING risk-ranked files in full and reasoning from first principles, not by grepping for known patterns. Catches the long tail (project-specific wrappers, business logic, cross-function flaws) that pattern-gated producers structurally miss. Token-expensive; budget-bounded and risk-ranked. Promotes leads into .kuzushi/findings.json (source "deep-scan").

2026-06-110

taint-analysis

allsmog/kuzushi-security-plugin

IRIS-style source→sink taint hunt. Ranks a typed CWE catalog for the repo, then drives subagents to label dangerous sinks, label sources of user input, run Joern/CodeQL dataflow queries (or same-file linking) to connect them, and triage each flow as finding/candidate/rejected. Promotes verdicts into .kuzushi/findings.json. Benefits from a prebuilt CodeQL DB / Joern CPG but degrades gracefully.

2026-06-110

threat-hunt

allsmog/kuzushi-security-plugin

Adversarial per-threat review (Carlini doctrine). For each threat in .kuzushi/threat-model.json, state attacker capabilities, walk source→sink, attempt to bypass every guard, and assign a verdict with evidence. Promotes findings into .kuzushi/findings.json. Requires /threat-model first.

2026-06-110

partition

allsmog/kuzushi-security-plugin

Split the attack surface into parallel-discovery partitions so fan-out hunters explore different subsystems instead of converging on the same shallow bug. Deterministic grouping of /x-ray entry points by component → .kuzushi/partitions.json, which a hunt coordinator hands to one subagent per partition. Run /x-ray first.

2026-06-100

poc

allsmog/kuzushi-security-plugin

Empirical proof-of-concept for the PoC-ready findings. For each finding /verify marked confirmed-exploitable or inconclusive, synthesize a minimal harness that triggers the bug; a host script then runs it in a sandbox (Docker --network none when present, else a gated local run) and classifies the crash into a proof verdict. Attaches a poc block onto each finding. Requires /verify first.

2026-06-100

name	benchmark
description	Measure recall / precision / false-proof rate of the pipeline against a ground-truth manifest. Scores either the bundled planted-vulnerability corpus (regression) or a live run's findings.json against a manifest you supply. Deterministic — no agent, no network. Use to prove a change to the producers helps rather than hurts.
context	inline
user-invocable	true

Benchmark

Run it

Bundled corpus (regression): node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" scores every case under bench/cases/ using its recorded findings.json. Add --case <name> for one case.
A live run: node "${CLAUDE_PLUGIN_ROOT}/scripts/cmd/benchmark.mjs" --target "<repo>" --ground-truth "<manifest.json>" scores <repo>/.kuzushi/findings.json after you've run the pipeline.

Ground-truth manifest

Reading the result

When NOT to use

To find bugs — /benchmark measures a run that already happened; it runs no analysis.
On a target with no ground-truth manifest — without annotated expectations there is nothing to score against; write the manifest first (or use the bundled corpus).
As a security verdict — a green benchmark means the corpus didn't regress, not that a given repo is bug-free.

Rationalizations to Reject

"Recall is 1.0 on the corpus, so the tool finds everything." → The corpus is a tiny planted set; it proves non-regression on known shapes, not coverage of the unknown. Grow the corpus when you confirm a new bug class, or the number lies by omission.
"Precision dipped but the new hits look plausible." → Plausible ≠ correct. A dip means a decoy got flagged; open it and confirm the guard before accepting the change.
"falseProofs went up but the bugs are real." → A false proof is a proven hit on a decoy — the harness fired on a non-bug. That's a soundness break regardless of intent; fix the harness/differential gate before shipping.