validate-on-project

name	validate-on-project
description	Use to validate ALL Scalpel capabilities on a real-world Python project — static analysis accuracy, genome building, reachability classification, pruning safety, and context generation. Invoke with: /validate-on-project <project-url-or-path>
allowed-tools	Bash, Read, Write, Edit, Grep, Glob, Agent

Validate on Project — Full Pipeline Validation

Purpose

Comprehensive end-to-end validation of Scalpel's entire pipeline against a real Python project. Tests EVERY major capability — not just pruning, but analysis accuracy, genome quality, classification correctness, and context output. Produces a detailed report card.

When to Use

• After significant changes to any Scalpel crate
• Before releases or major milestones
• To benchmark Scalpel's accuracy on a new codebase
• When the user provides a Python project and says "test this" or "validate this"

Input

/validate-on-project https://github.com/org/repo.git
/validate-on-project /path/to/local/python/project

Overview

The validation runs 6 test suites in sequence. Each produces a PASS/FAIL with metrics.

Suite 1: Setup .............. clone, venv, install, baseline tests
Suite 2: Static Analysis .... function extraction, call sites, imports
Suite 3: Genome Quality ..... nodes, edges, edge resolution rate
Suite 4: Classification ..... reachability, false positive spot-checks
Suite 5: Prune Safety ....... stub dead code, rerun tests, zero regressions
Suite 6: Context Output ..... MCP tools produce valid, useful output

---

Suite 1: Setup

1a. Clone or Locate

# If URL:
git clone --depth 1 <url> /tmp/scalpel-validate-project
PROJECT_DIR="/tmp/scalpel-validate-project"

# If local path:
PROJECT_DIR="<path>"

1b. Detect Project Structure

Find these automatically:

# Project name
PROJECT_NAME=$(grep '^name' $PROJECT_DIR/pyproject.toml 2>/dev/null | head -1 | sed 's/.*= *"\(.*\)"/\1/')

# Library source directory (heuristic: directory matching project name, or src/)
LIB_DIR=$(ls -d $PROJECT_DIR/*/ | grep -v tests | grep -v docs | grep -v __pycache__ | grep -v '^\.' | head -1)

# Test directory
TEST_DIR="$PROJECT_DIR/tests"
[ -d "$TEST_DIR" ] || TEST_DIR="$PROJECT_DIR/test"

# Python version
for v in python3.13 python3.12 python3.11 python3.10; do
  command -v $v &>/dev/null && PYTHON=$v && break
done

Report detected structure to user. Ask for confirmation if ambiguous.

1c. Create Venv and Install

$PYTHON -m venv /tmp/scalpel-validate-venv
VENV_PIP="/tmp/scalpel-validate-venv/bin/pip"
VENV_PYTHON="/tmp/scalpel-validate-venv/bin/python"

# Try install strategies in order
$VENV_PIP install -e "$PROJECT_DIR[all]" 2>/dev/null \
  || $VENV_PIP install -e "$PROJECT_DIR[dev,test]" 2>/dev/null \
  || $VENV_PIP install -e "$PROJECT_DIR" 2>/dev/null

$VENV_PIP install pytest pytest-timeout

Handle ModuleNotFoundError during test collection: install missing package and retry (up to 5 rounds).

1d. Baseline Tests

cd $PROJECT_DIR
$VENV_PYTHON -m pytest $TEST_DIR -q --timeout=60 -p no:sugar 2>&1

GATE: Baseline must have >90% pass rate. Record exact passed/failed/skipped counts.

Suite 1 Metrics:

Python version: <version>
Library dir: <path>
Test dir: <path>
Baseline: <N> passed, <N> failed, <N> skipped

---

Suite 2: Static Analysis

Run Scalpel's Python static analyzer on the library source AND tests.

Rust Test Harness

Write a temporary test file tests/_validate_project_test.rs that calls the analyzer APIs directly. This is needed because the CLI is JS-focused.

let lib_result = scalpel_analyze::python::analyze_py_directory(lib_dir).unwrap();
let test_result = scalpel_analyze::python::analyze_py_directory(test_dir).unwrap();

Metrics to Capture

Metric	What to Check	Pass Criteria
Functions extracted	lib_result.functions.len()	> 50 for any real project
Call sites	lib_result.calls.len()	> 50
Imports	lib_result.imports.len()	> 10
caller_name coverage	% of calls with non-empty caller_name	> 70%
Analysis time	Wall clock	< 10s for 100K lines

Caller Name Spot-Check

Pick 5 random call sites with caller_name set. Verify each:

grep -n "<callee_name>" <caller_file>

Confirm the call is actually inside the named caller function.

Suite 2 Metrics:

Library functions: <N>
Test functions: <N>
Call sites: <N> (<N>% with caller_name)
Imports: <N>
Analysis time: <N>s
Caller spot-check: <N>/5 correct

---

Suite 3: Genome Quality

Build the Code Genome from the combined static analysis results.

let mut genome = scalpel_core::merge::build_static_genome(&combined, project_root);

Metrics to Capture

Metric	What to Check	Pass Criteria
Node count	genome.node_count()	Close to total functions
Edge count	genome.edge_count()	> 0 (ideally > nodes * 0.1)
Edge resolution rate	edges / call_sites	> 5% (higher = better)
Orphan nodes	Nodes with 0 incoming + 0 outgoing edges	< 50% of nodes

Edge Spot-Check

Pick 3 edges and verify they make sense:

for (_, src, tgt, _) in genome.edges().take(3) {
    let src_name = genome.resolve(genome.get_node(src).unwrap().name);
    let tgt_name = genome.resolve(genome.get_node(tgt).unwrap().name);
    // Verify src actually calls tgt via grep
}

Suite 3 Metrics:

Genome nodes: <N>
Genome edges: <N>
Edge resolution rate: <N>% (<N> edges / <N> call sites)
Orphan nodes: <N> (<N>%)
Edge spot-check: <N>/3 correct

---

Suite 4: Classification

Mark entrypoints and run reachability classification.

Entrypoint Rules (MANDATORY — all 5)

1. Test functions — name.starts_with("test_")
2. __init__.py functions — file.contains("__init__")
3. Class methods — name.contains('.') (instance dispatch invisible to static analysis)
4. Classes — first char uppercase (framework instantiation)
5. Module-level call targets — calls with empty caller_name (import side effects)

genome.classify_all();

Metrics

Metric	Pass Criteria
Reachable %	> 50% of nodes (if lower, entrypoint marking is wrong)
Unreachable %	Reasonable — varies by project

False Positive Spot-Check

Pick 5 functions from the unreachable list. For each:

grep -rn "<function_name>" <library_source>/

Classify:

• True positive: only appears at its def line — genuinely dead
• False positive: appears at a call site or as a value reference — incorrectly classified

Suite 4 Metrics:

Entrypoints marked: <N> (+ <N> module-level targets)
Reachable: <N> (<N>%)
Unreachable: <N> (<N>%)
Spot-check: <N>/5 true positives, <N>/5 false positives
False positive rate (spot-check): <N>%

---

Suite 5: Prune Safety

The ultimate test: stub unreachable functions, run the full test suite.

Safety Filters (MANDATORY — all 4)

Before pruning, filter candidates:

1. Library-only — never prune test files
2. No __init__.py — public API surface
3. No dunder methods — protocol methods
4. Value-reference scan — word-boundary text search for each candidate name; if referenced beyond its def, skip it

Prune + Retest

1. Copy project to temp dir
2. Stub candidates with scalpel_prune::stub_py_functions
3. Install pruned copy in venv
4. Run full test suite
5. Restore original

Verdict Rules

• SAFE: test diff <= 2 AND zero scalpel: function RuntimeErrors
• UNSAFE: test diff > 2 OR any scalpel RuntimeError

If UNSAFE, list each failing test + the function that was incorrectly pruned.

Suite 5 Metrics:

Candidates before filters: <N>
Skipped (value-referenced): <N>
Final prunable: <N> in <N> files
Functions stubbed: <N>
Baseline tests: <N> passed
Pruned tests: <N> passed (diff: <N>)
Scalpel RuntimeErrors: <N>
Verdict: SAFE / UNSAFE

---

Suite 6: Context Output

Validate that Scalpel's MCP tools produce useful output from the genome.

Tests

If the genome was exported (via cargo run -p scalpel-cli -- export):

# Test JSON export validity
python3 -c "import json; json.load(open('.scalpel/genome-export.json'))"

# Test genome summary via MCP
# (if MCP server is available)

If MCP tools are available, validate:

• get_genome_summary() returns non-empty data
• search_genome("<known_function>") finds the function
• get_unused_code() returns the unreachable list
• get_dependencies("<function>") returns edges

If MCP is not available, verify programmatically:

// Genome stats
let stats = genome.stats();
assert!(stats.node_count > 0);
assert!(stats.edge_count > 0);

// Search works
let results = genome.find_nodes_by_suffix("some_known_function");
assert!(!results.is_empty());

Suite 6 Metrics:

JSON export: valid / invalid
Genome summary: <N> nodes, <N> edges
Search: <N> queries, <N> hits
Dependencies: <N> queries, <N> edges returned

---

Final Report Card

After all 6 suites, produce a consolidated report:

╔══════════════════════════════════════════════════╗
║     SCALPEL VALIDATION REPORT                    ║
║     Project: <name>                              ║
║     Date: <date>                                 ║
╠══════════════════════════════════════════════════╣
║  Suite 1: Setup .................... PASS / FAIL ║
║  Suite 2: Static Analysis ......... PASS / FAIL ║
║  Suite 3: Genome Quality .......... PASS / FAIL ║
║  Suite 4: Classification .......... PASS / FAIL ║
║  Suite 5: Prune Safety ............ PASS / FAIL ║
║  Suite 6: Context Output .......... PASS / FAIL ║
╠══════════════════════════════════════════════════╣
║  Overall: <N>/6 PASSED                           ║
╚══════════════════════════════════════════════════╝

Key Numbers:
  Functions: <N> extracted, <N> nodes, <N> edges
  Classification: <N>% reachable, <N>% unreachable
  Pruning: <N> stubbed, <N>/<N> tests pass
  False positive rate: <N>% (spot-check)

Cleanup

rm -rf /tmp/scalpel-validate-project /tmp/scalpel-validate-pruned /tmp/scalpel-validate-venv
rm -f tests/_validate_project_test.rs

Known Good Projects for Validation

These are well-tested Python projects suitable for validation:

Project	Stars	Tests	Good For
fastapi/fastapi	85K+	3135	Web framework, decorators, async
pallets/flask	69K+	500+	Web framework, extensions
psf/requests	52K+	400+	HTTP client, simple structure
encode/httpx	13K+	800+	Async HTTP, modern Python
pydantic/pydantic	22K+	2000+	Data validation, complex types

Clone with git clone --depth 1 to save time and disk.