scalpel-verify

name	scalpel-verify
description	Use after ANY change to analysis, merge, classification, or pruning logic. Also use before claiming accuracy improved or false positives fixed. Mandatory exit gate for all domain skills.
allowed-tools	Bash, Read, Grep

Scalpel Verification — Accuracy Discipline

Core Principle

Define what "correct" means BEFORE measuring. Measure with Scalpel's own tools. Compare against a recorded baseline. If you have three different numbers from the same codebase, your methodology is broken — not your code.

Violating the letter of this rule is violating the spirit of this rule.

When to Use

• After ANY change to scalpel-analyze (parsing, call extraction, node type filtering)
• After ANY change to scalpel-core (merge logic, confidence classification, reachability BFS)
• After ANY change to scalpel-prune (safety gates, re-export chains, SCC handling)
• Before writing "accuracy improved" anywhere — in code, comments, or conversation
• Before writing "fixed false positives" anywhere
• This skill is the mandatory exit gate after using any other scalpel domain skill

HARD GATE

DO NOT claim accuracy improved without completing all 6 steps below.
DO NOT claim false positives are fixed without completing all 6 steps below.
DO NOT skip steps because "the change was small" or "it's obvious it worked."

The six steps are a unit. Completing five is not completing six.

Domain Model — MCP Tools

This skill uses Scalpel's own MCP tools to verify Scalpel's own accuracy (dogfooding). Each tool has a specific verification role:

MCP Tool	Verification Use
search_genome(query)	Check if a specific function is correctly classified; confirm a formerly-false-positive is now gone
get_unused_code()	Get the full unreachable list; compare count against baseline
trace_call_chain(from, to)	Verify cross-crate call edges resolve correctly after merge changes
get_genome_summary()	Get total node/edge counts; detect unexpected graph size changes
get_function_profile(name)	Inspect confidence level + runtime metrics for a specific function
reload_genome()	Force re-read from disk after re-analysis; MUST be called before any other query

Critical: reload_genome() must be called before any query after re-running the pipeline. Without it, all MCP tool results reflect stale data.

Process — Mandatory 6-Step Sequence

Step 1: Define success criterion (BEFORE making changes)

Write it down explicitly before touching any code:

"After this change, <condition> should change from <before> to <after>."

Examples:

• "After filtering type definitions from node extraction, unreachable count should drop from 245 to under 50."
• "After fixing the import index lookup, cross-crate call edges for scalpel-core → scalpel-proto should increase from 0 to >10."
• "The function format_bytes should no longer appear in get_unused_code() output."

No criterion written = no verification possible. Stop and write one.

Step 2: Record baseline (BEFORE making changes)

# Delete any stale export first
rm -f .scalpel/genome-export.json

# Record current state via MCP
reload_genome()
get_genome_summary()        # note: total nodes, total edges, unreachable count
get_unused_code()           # save full list

Record at minimum:

• Total node count
• Total edge count
• Unreachable node count
• 3–5 specific functions you expect the change to affect

If no baseline exists (first run), establish one now before changing anything.

Step 3: Re-run full pipeline (AFTER making changes)

# Mandatory: delete cached export or you get stale results
rm -f .scalpel/genome-export.json

# Re-run trace + genome generation
cargo run -p scalpel-cli -- trace --lang rust -- cargo test --all
cargo run -p scalpel-cli -- genome .scalpel/rust-cov

Do not skip rm -f .scalpel/genome-export.json. This is the single most common cause of false "it worked" conclusions.

Step 4: Measure against success criterion

reload_genome()                              # MUST be first
get_genome_summary()                         # compare node/edge counts to baseline
get_unused_code()                            # compare unreachable count to baseline
search_genome("<specific_function>")         # verify expected classification change
get_function_profile("<specific_function>")  # inspect confidence + metrics

Check each part of your Step 1 criterion:

• Did the number change in the expected direction?
• Did the specific functions you expected to change actually change?
• Did any counts change unexpectedly (potential regression)?

Step 5: Spot-check 3 items from the unreachable list

Pick 3 items from get_unused_code() output — do NOT only pick ones you expect to be correct. Include at least one you are uncertain about.

For each:

grep -rn "function_name" crates/

Then determine:

• Is the grep hit a definition only, with no call sites? → True positive. Confirmed dead.
• Is there at least one call site in the grep output? → False positive. The genome missed an edge. Investigate why before committing.

The grep step breaks the self-referential circularity of using Scalpel to verify Scalpel. It is not optional.

Step 6: Report with numbers

Every accuracy claim must include all of the following:

Unreachable: <baseline> → <current> (<current_percentage>% of total nodes)
Spot-checked: 3 items — [true/false] positive, [true/false] positive, [true/false] positive
Known false positives: <count> — [list each with root cause or "none"]
Precision: <confirmed_true_positives> / <total_unreachable>
Pipeline run: <timestamp or commit>

Example of acceptable reporting:

"Unreachable: 245 → 37 (3.4% of 1076 nodes). Spot-checked 3: all true positives. Known false positives: 0. Precision: not fully measured but no new FPs found in spot-check. Pipeline run: after filtering type-def nodes from extraction."

Example of unacceptable reporting:

"Accuracy improved significantly." ← no baseline, no numbers, not verifiable

Known Traps

1. Cached genome — run_genome and all MCP tools read .scalpel/genome-export.json if it exists on disk. If you skip rm -f .scalpel/genome-export.json before re-running the pipeline, every MCP query returns stale pre-change results. You will conclude your change worked when you measured nothing.

2. Grep methodology confusion — grep -rn "function_name" crates/ finds both the function declaration and any call sites. A result with only the definition line is a true positive (dead code). A result with call sites is a false positive. Always check whether each grep hit is a definition or a call.

3. Accuracy denominator — "27 false positives" means nothing without knowing the total unreachable count. "27 out of 37 unreachable" means Scalpel is 27% accurate. "27 out of 1076 nodes" means 2.5% false positive rate on the full graph. Always report both the raw number and the percentage with explicit denominator.

4. Self-referential bias — Scalpel reporting on its own accuracy has inherent circularity: if the genome has a bug, the bug affects the verification query. Step 5 (raw grep spot-check) is the circuit-breaker. Never skip it on the grounds that "the MCP tools say it's fine."

5. Synthetic vs real fixtures — Unit tests with 5-function code snippets pass even when real codebase analysis is broken. The validation surface for self-analysis is 72 source files, 1076 nodes, 8 crates with cross-crate edges. Always run self-analysis; synthetic fixture tests are necessary but not sufficient.

Red Flags — STOP if you think any of these

Thought	What It Means
"The change was small, verification is overkill"	Small changes cause subtle regressions. Run it.
"I can tell from the code it's correct"	Code correctness ≠ accuracy on real data. Measure.
"Accuracy improved" (without having run steps 1–6)	This is a claim without evidence. Run the steps.
"The unit tests pass so it's good"	Synthetic fixtures don't cover cross-crate edge resolution at scale.
"I'll record the baseline after the change"	You cannot compare before/after without a before.
"The MCP tools confirm it"	Without reload_genome() first, MCP tools are showing you the old genome.
"Spot-checking seems like extra work"	Spot-check is the only step that catches self-referential measurement errors.
"I checked 3 items I already knew were correct"	That is confirmation bias, not verification. Pick uncertain items.

Common Rationalizations

Rationalization	Why It Fails
"I'm confident in the fix"	Confidence is not measurement.
"Just this once, I'll skip baseline"	You cannot compute delta without a start point.
"The test suite covers it"	Test suite catches bugs; accuracy verification catches correctness-at-scale.
"Different wording, so the rule doesn't apply"	Spirit over letter. If you're claiming improvement, run the steps.
"I deleted the genome file, that counts as baseline"	Deleting is not recording. Baseline = run get_genome_summary() and save the numbers.
"reload_genome was called earlier in this session"	Earlier calls don't count. Call it again after re-running the pipeline.

Integration With Other Domain Skills

scalpel-verify is the universal exit gate. Every other domain skill's verification protocol is a subset of this one:

/scalpel:static-analysis → make changes → /scalpel:verify (mandatory)
/scalpel:graph-engine    → make changes → /scalpel:verify (mandatory)
/scalpel:tracing         → make changes → /scalpel:verify (mandatory)
/scalpel:pruning         → make changes → /scalpel:verify (mandatory)
/scalpel:ipc-transport   → make changes → cargo test -p scalpel-ipc (sufficient for IPC)

IPC/transport changes are the exception: they affect measurement reliability, not classification accuracy. For those, the crate-level test suite and stress test (1M events, zero loss) are the verification gate.

Quick Reference

# Full verification run — copy/paste block
rm -f .scalpel/genome-export.json
cargo run -p scalpel-cli -- trace --lang rust -- cargo test --all
cargo run -p scalpel-cli -- genome .scalpel/rust-cov

# MCP sequence after pipeline completes
reload_genome()
get_genome_summary()
get_unused_code()
search_genome("<function_you_changed>")
get_function_profile("<function_you_changed>")

# Spot-check a candidate from get_unused_code()
grep -rn "<candidate_function>" crates/

Baseline numbers to beat (recorded during last self-analysis):

• Unreachable: 37 nodes (3.4% of 1076 total)
• Total nodes: 1076
• Total edges: 1560
• Known false positives at that baseline: 0

If your change increases unreachable count above 37 without an explanation, investigate each new item before committing.