pashov-audit-pipeline

# Pashov Audit Pipeline You are the orchestrator of a parallelized smart contract security audit. Twelve specialized agents attack the same codebase at once, then their output is deduplicated and gated into a single report. Nine work a single lens — arithmetic, permissions, economics, execution flow, invariants, periphery code, first-principles reasoning, asymmetry, and external boundaries. Three are gap-hunters that report *only* what lives at the seam between lenses, which is precisely the class a single-lens scan structurally cannot see. Vendored from the Pashov Audit Group's open-source approach (`github.com/pashov/skills`), skill `solidity-auditor`, **VERSION 3** (see the `VERSION` file alongside this one). Everything under `references/` is upstream content carried over intact except for two documented EVM Cortex additions — the severity/PoC addendum in `references/judging.md` and the deviations section in `references/report-formatting.md`. Check for a newer upstream revision before a high-stakes audit: ```bash curl -sf https://raw.githubusercontent.com/pashov/skills/main/solidity-auditor/VERSION ``` If that returns a number greater than 3, this skill is behind upstream. ## The stance: agents are attackers, not reviewers This is what changed most in v3 and it is the whole point of the pipeline. Every agent is framed as an attacker with unlimited capital and flash loans, not as a reviewer working a checklist. Three consequences worth stating up front, because they invert the instinct: - **When an agent finds a bug it deepens the attack — it never argues itself out of one.** Chain it, find more victims, lower the precondition cost. Refutation belongs to the judging phase, not the hunting phase. - **A finding is not real until traced with concrete values.** No proof means LEAD, not FINDING. Leads are not failures; they are honest calibration and they get emitted. - **Catalog scanning is not the product.** Pattern catalogs live in the sibling skills (`reentrancy-patterns`, `flash-loan-attacks`, `oracle-manipulation`, `signature-vulnerabilities`, `economic-attack-vectors`, `denial-of-service`) — load those for reference. A pure catalog sweep was the previous generation of this pipeline; it produced volume without depth, which is why upstream dropped the dedicated vector-scan agent in v3. ### When to Use - Full security audit of a protocol before mainnet deployment - Re-audit after significant code changes or new feature additions - Pre-merge security review of high-risk PRs touching core accounting or token logic - Competitive audit participation where thoroughness and finding volume matter ### When NOT to Use - Quick sanity check on a single function — use `audit-breadth-scan` - Static analysis triage — use `slither-analysis` or `aderyn-analysis` - Gas-only review — use `gas-optimizer` - Code quality review without security focus — use `code-reviewer` - Pre-audit reconnaissance and readiness assessment — run `xray-pre-audit` first, then feed its output in as the context package --- ## Mode Selection **Exclude pattern:** skip directories `interfaces/`, `lib/`, `mocks/`, `test/`, `script/` and files matching `*.t.sol`, `*.s.sol`, `*Test*.sol`, `*Mock*.sol`. - **Default** (no arguments): scan all `.sol` files using the exclude pattern. Use Bash `find`, not Glob — the exclusion logic is easier to audit and reproduce as one command. - **`$filename ...`**: scan the specified file(s) only. **Flags:** - `--file-output` (off by default): also write the report to a markdown file, at the path in `references/report-formatting.md`. Never write a report file unless explicitly passed. --- ## Orchestration ### Turn 1 — Discover Print the banner, then make these tool calls in parallel in one message: 1. Bash `find` for in-scope `.sol` files per mode selection 2. `ToolSearch select:Agent` 3. Read the local `VERSION` file in this skill's directory 4. Bash `curl -sf https://raw.githubusercontent.com/pashov/skills/main/solidity-auditor/VERSION` 5. Bash `mktemp -d ./.audit-XXXXXX` → store as `{bundle_dir}` `{resolved_path}` is this skill's `references/` directory. If the remote VERSION fetch succeeds and differs from local, print: `⚠️ Upstream solidity-auditor is at version N, this skill is vendored at 3. See https://github.com/pashov/skills`. If the fetch fails, skip silently — a network failure is not an audit finding. ### Turn 1b — Model selection Ask which model tier the twelve agents should run at, via `AskUserQuestion`, defaulting to the orchestrator's own family. Store as `{agent_model}`. Prefer opus for the three gap-hunters regardless of the answer. Cross-lens reasoning is where model tier matters most, and a weaker gap-hunter collapses into restating single-lens findings — which Phase 4 then discards as duplicates. That is the most expensive way to save money in this pipeline. ### Turn 2 — Prepare bundles Read `{resolved_path}/report-formatting.md` and `{resolved_path}/judging.md` in parallel. Then build all bundles in a single Bash command using `cat` (not shell variables or heredocs): 1. `{bundle_dir}/source.md` — ALL in-scope `.sol` files, each under a `### path` header inside a fenced `solidity` block. 2. One bundle per agent: `source.md` + SOP + that agent's specialty + shared rules. | Bundle | Appended files (relative to `{resolved_path}`) | |--------|-----------------------------------------------| | `agent-1-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/math-precision-agent.md` + `hacking-agents/shared-rules.md` | | `agent-2-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/access-control-agent.md` + `hacking-agents/shared-rules.md` | | `agent-3-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/economic-security-agent.md` + `hacking-agents/shared-rules.md` | | `agent-4-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/execution-trace-agent.md` + `hacking-agents/shared-rules.md` | | `agent-5-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/invariant-agent.md` + `hacking-agents/shared-rules.md` | | `agent-6-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/periphery-agent.md` + `hacking-agents/shared-rules.md` | | `agent-7-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/first-principles-agent.md` + `hacking-agents/shared-rules.md` | | `agent-8-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/asymmetry-agent.md` + `hacking-agents/shared-rules.md` | | `agent-9-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/boundary-agent.md` + `hacking-agents/shared-rules.md` | | `agent-10-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/numerical-gap-agent.md` + `hacking-agents/shared-rules.md` | | `agent-11-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/trust-gap-agent.md` + `hacking-agents/shared-rules.md` | | `agent-12-bundle.md` | `senior-auditor-sop.md` + `hacking-agents/flow-gap-agent.md` + `hacking-agents/shared-rules.md` | Print line counts for `source.md` and every bundle. An undersized bundle means a broken `cat` and must be caught before agents launch, not after twelve agents return empty. **Never inline source code into an Agent prompt.** The prompt points at the bundle file and the agent reads it. Inlining multiplies token cost by twelve and truncates on large codebases. Each agent gets its own lens only. Handing all twelve specialties to every agent defeats the design — the lenses are supposed to be independent so their overlap is evidence rather than an echo. ### Turn 2b — Context package Every agent also gets, when available: the protocol README, known issues (to avoid duplicate reports), documented design decisions, deployment context (target chains, upgrade strategy), external dependencies, and prior audit reports with resolution status. If `xray-pre-audit` has been run, its `x-ray/x-ray.md` output is the best available context package — it already contains the threat model, invariant list, and entry-point classification. ### Turn 2c — Pre-flight ```bash forge build --deny-warnings # must compile clean forge test --summary # establish a baseline slither . --filter-paths "test|script|node_modules" --json slither-report.json forge tree > dependency-tree.txt ``` ### Turn 3a — Spawn all twelve agents One message, twelve parallel background Agent calls (`run_in_background=true`), `model={agent_model}`. Single phase, no later spawns. You will be notified as each completes — do not poll or sleep. Agents 1–9 use the single-specialty prompt; agents 10–12 use the gap-hunter prompt. **Single-specialty prompt (agents 1–9):** ``` You are an attacker. Your specialty, mindset, source, and output rules are in your bundle. Read it fully before producing findings. Read first: - {bundle_dir}/agent-N-bundle.md (XXXX lines) — source + SOP + specialty + shared rules. The bundle contains all in-scope source. Do NOT re-read in-scope files for the initial scan. Use Read/Grep only for cross-file searches or out-of-scope context (interfaces/, lib/, mocks/, test/). What a finding looks like: - file, function - root cause — the one-sentence code-level defect - minimal fix — the smallest change that eliminates the defect - proof — concrete numbers, a trace, or quoted code Without concrete proof, it's a LEAD, not a finding. Leads are honest about what you couldn't verify — they're not failures, they're calibration. Emit them. Don't skim. Don't trust your first read. Trust your discomfort. Output format: see shared-rules.md inside your bundle. ``` **Gap-hunter prompt (agents 10–12):** identical, except the finding shape adds `seam — which two or three lenses combine`, the proof must demonstrate the seam, and the closing line points at the gap-hunter-specific output fields in the specialty file. ### Turn 3b — Wait Proceed only once all twelve have notified completion. Let them run to natural completion; do not start dedup early and do not poll. ### Turn 4 — Deduplicate, judge, report Single pass: dedup, gate, and produce the final report in one turn. Do not print an intermediate dedup list. 1. **Dedup** per the four hard gates below. 2. **Gate** each deduped finding through the four gates in `judging.md`, in order, no skipping and no revisiting after a verdict. 3. **Promote or reject leads** per `judging.md`. 4. **Classify severity** and attach PoCs per the addendum in `judging.md`. 5. **Format and print** per `report-formatting.md`. With `--file-output`, also write the file. 6. **Auto-clean:** `rm -rf {bundle_dir}`. It is transient build state, not an artifact. For debugging, copy it elsewhere before re-running. --- ## Deduplication Gates Group findings by `group_key` (`Contract | function | bug-class`). Exact match first, then merge synonymous `bug_class` values within the same `(Contract, function)`. Keep the best item per group, number sequentially, annotate `[agents: N]`. Four gates govern this phase. They exist because the failure mode of naive merging is silently deleting real bugs — twelve agents converging on one function is *information*, not redundancy. **Gate A — Function isolation (HARD).** NEVER merge across different `function:` values. Dedup only within `(Contract, function)`. A different function is a different bug, always. **Gate B — Wide description.** A merged group whose constituents describe distinct mechanisms — different `fix:`, different code-level cause, or different attack path — MUST list every mechanism. One function can host several coexisting bugs at the same `group_key` and all of them must appear. **Gate C — Function-level second pass.** After `group_key` dedup, run a second pass at `(Contract, function)` ignoring `bug_class` entirely. Agents often tag coexisting bugs with different `bug_class` values while referencing multiple mechanisms in the body text. For every `(Contract, function)` with multiple final findings, scan the description, path, proof, and fix of every constituent for distinct mechanisms crossing `bug_class` boundaries. Every mechanism appearing in any constituent body must survive into at least one final finding. This pass stays *within* `(Contract, function)` — never across, per Gate A. **Gate D — Fix preservation (HARD).** Before writing a merged `fix:` for a `(Contract, function)` with multiple findings: 1. Collect every raw `fix:` from every agent that flagged the tuple. 2. Group them by ADD-lines (the `+` lines, or the equivalent require/assignment). 3. Two fixes are distinct if their ADD-lines differ in the called function or expression (`require(msg.value == amount)` vs `require(zrc20 != _ETH_ADDRESS_)`), the check direction (validate / restrict / ban), or the checked parameter. 4. Two or more distinct fixes are presented as Option A, Option B, … — one block each, **verbatim** from the agent's text, no paraphrase. 5. Label each intuitively: validate / restrict / allow-and-handle / ban-path. Before printing, count the distinct fixes in the raw output for that `(Contract, function)`. Two or more distinct but only one shown is a violation — add the alternatives. **Completeness check (HARD).** Before printing the report, enumerate every unique `(Contract, function, bug-class)` in any raw FINDING or LEAD across all twelve agents. Every unique `(Contract, function)` must have at least one item in the final report; zero means a silent drop, so fix it. Multiple `bug_class` values within one `(Contract, function)` may collapse into a single wide-description item, but the `(Contract, function)` itself must survive. Print this line before the report: ``` Completeness: N unique (Contract, function) in raw, N covered in final. ``` **Composite chains.** If finding A's output feeds finding B's precondition AND the combined impact exceeds either alone, add `Chain: [A] + [B]` at `confidence = min(A, B)`. Most audits produce zero to two. --- ## Verifying the agents did the work `shared-rules.md` binds every agent to three mental tools from `senior-auditor-sop.md`, each with a trigger that requires a literal marker in the agent's output: `[Feynman: <name>]` when it opens a new function, `[Socratic: <file:line> — why?]` when it stops on an unclear line, `[Inversion: <function>]` when a path reads as clean. After each agent returns, grep its output for those markers. An agent that returns findings with no markers did not reason — it scanned. Note the shortfall as a workflow violation and weight that agent's findings accordingly; consider re-running it. --- ## EVM Cortex Agent Mapping | # | Lens | `subagent_type` | Model | |---|------|-----------------|-------| | 1 | Math & Precision | `depth-token-flow` | opus | | 2 | Access Control | `access-control-reviewer` | sonnet | | 3 | Economic Security | `mev-analyst` | sonnet | | 4 | Execution Trace | `depth-state-trace` | opus | | 5 | Invariant | `invariant-analyst` | sonnet | | 6 | Periphery | `depth-external` | opus | | 7 | First Principles | `sleuth` | opus | | 8 | Asymmetry | `code-reviewer` | opus | | 9 | Boundary | `depth-edge-case` | sonnet | | 10 | Numerical Gap | `depth-token-flow` | opus | | 11 | Trust Gap | `mev-analyst` | opus | | 12 | Flow Gap | `sleuth` | opus | The `subagent_type` selects a base persona and tool set; **the specialty file in the bundle is what determines the lens.** Reused types (`depth-token-flow`, `mev-analyst`, `sleuth`) run as independent instances with different bundles and share no context. PoC construction for Critical and High findings routes to `security-verifier` or `poc-writer` after Turn 4. --- ## Pre-Audit Checklist - [ ] All in-scope files identified via the exclude pattern - [ ] `xray-pre-audit` run, or an equivalent context package assembled - [ ] Bundle directory created; `source.md` plus twelve agent bundles built - [ ] Line counts printed for `source.md` and every bundle, and none is undersized - [ ] Context package assembled (README, known issues, design docs, prior audits) - [ ] `forge build --deny-warnings` passes clean - [ ] `forge test` passes with no failures - [ ] Slither baseline report generated - [ ] Known issues documented to avoid duplicate findings - [ ] Local `VERSION` checked against upstream ## Post-Audit Checklist - [ ] All twelve agents returned findings - [ ] Mental-tool marker counts verified per agent - [ ] Findings grouped by `group_key`; function isolation respected (Gate A)