prune-safety-test

name	prune-safety-test
description	Use to validate Scalpel's pruning safety on any Python project. Analyzes a real-world Python project with test suite, identifies dead code, stubs it, then runs the full test suite to prove nothing broke. Invoke with: /prune-safety-test <project-url-or-path>
allowed-tools	Bash, Read, Write, Edit, Grep, Glob, Agent

Prune Safety Test — Real-World Pruning Validation

Purpose

End-to-end validation that Scalpel can safely identify and remove unused Python code without breaking a project's test suite. This is the ultimate proof: analyze, prune, test, compare.

When to Use

• After changes to scalpel-analyze (Python parser, call extraction)
• After changes to scalpel-prune (py_rewriter, safety gates)
• After changes to scalpel-core (merge logic, reachability, classification)
• To validate Scalpel against a new real-world Python project
• Before claiming "pruning is safe" — run this on a real project first

Input

The skill takes one argument: a Python project path or git URL.

/prune-safety-test https://github.com/fastapi/fastapi.git
/prune-safety-test /path/to/local/python/project

Requirements for the target project:

• Python source code in a recognizable library directory
• A tests/ directory with pytest-compatible test cases
• A pyproject.toml or setup.py for installation

Process — 7 Phases

Phase 1: Setup

1a. Clone or locate the project

# If URL provided:
git clone --depth 1 <url> /tmp/scalpel-test-project
PROJECT_DIR="/tmp/scalpel-test-project"

# If path provided:
PROJECT_DIR="<path>"

1b. Detect project structure

Find these three things — they vary per project:

What	How to Find	Examples
Library source dir	Top-level directory matching project name, or src/	fastapi/, flask/, src/myproject/
Test directory	tests/ or test/	tests/, test/
Install extras	Check pyproject.toml for [project.optional-dependencies]	.[all], .[dev], .[test]

# Auto-detect library source directory
ls -d */ | grep -v tests | grep -v docs | grep -v __pycache__
# Check pyproject.toml for project name
grep "^name" pyproject.toml
# Check for optional deps
grep -A 5 '\[project.optional-dependencies\]' pyproject.toml

1c. Create venv and install

# Detect available Python >= 3.10
for v in python3.13 python3.12 python3.11 python3.10; do
  if command -v $v &>/dev/null; then PYTHON=$v; break; fi
done

# Create venv
$PYTHON -m venv /tmp/scalpel-test-venv

# Install project with all extras (try multiple strategies)
/tmp/scalpel-test-venv/bin/pip install -e ".[all]" 2>/dev/null \
  || /tmp/scalpel-test-venv/bin/pip install -e ".[dev,test]" 2>/dev/null \
  || /tmp/scalpel-test-venv/bin/pip install -e "." 2>/dev/null

# Install test runner
/tmp/scalpel-test-venv/bin/pip install pytest pytest-timeout

Handle missing deps: If pytest collection fails with ModuleNotFoundError, install the missing package and retry. Repeat up to 5 times. This is normal for projects with unpinned test deps.

Phase 2: Baseline Tests

Run the full test suite on the unmodified project.

cd $PROJECT_DIR
/tmp/scalpel-test-venv/bin/python -m pytest tests/ -q --timeout=60 -p no:sugar --ignore=tests/benchmarks 2>&1

Record:

• Total passed count (the number before "passed")
• Any skipped/xfailed counts
• Total runtime

HARD GATE: If baseline tests don't pass (>90% pass rate), STOP. The project's test suite must be healthy before we can validate pruning. Report the issue to the user.

Phase 3: Static Analysis + Genome

Write and run a temporary Rust integration test. This is necessary because the CLI prune command is JS-only; the Rust API supports Python.

Create the test file at tests/_prune_safety_test.rs:

use std::path::Path;

#[test]
fn prune_safety_test() {
    let src_dir = Path::new("<LIBRARY_SOURCE_DIR>");
    let test_dir = Path::new("<TEST_DIR>");

    // Analyze both library and tests
    let lib_result = scalpel_analyze::python::analyze_py_directory(src_dir).unwrap();
    let test_result = scalpel_analyze::python::analyze_py_directory(test_dir).unwrap();

    // Combine
    let mut combined = scalpel_analyze::js::StaticAnalysisResult::default();
    combined.functions.extend(lib_result.functions);
    combined.functions.extend(test_result.functions);
    combined.calls.extend(lib_result.calls);
    combined.calls.extend(test_result.calls);
    combined.imports.extend(lib_result.imports);
    combined.imports.extend(test_result.imports);

    // Build genome
    let mut genome = scalpel_core::merge::build_static_genome(
        &combined, Path::new("<PROJECT_DIR>")
    );

    // === ENTRYPOINT MARKING (critical for safety) ===
    // ... (see Phase 4 below)

    genome.classify_all();

    // === SAFETY FILTERS (critical for avoiding false positives) ===
    // ... (see Phase 5 below)

    // === PRUNING + TEST EXECUTION ===
    // ... (see Phase 6 below)
}

Report:

• Functions extracted (library + tests)
• Call sites found
• Genome nodes and edges

Phase 4: Entrypoint Marking

These rules are hard-won from real-world testing. Do NOT skip any.

Mark as entrypoints:

Rule	Why	Code
Test functions (test_*)	Tests are the usage roots	name.starts_with("test_")
__init__.py functions	Public API surface	file.contains("__init__")
Class methods (Name.method)	instance.method() invisible to static analysis	name.contains('.')
PascalCase names (classes)	Framework instantiation not in call graph	First char is uppercase
Module-level call targets	Side effects on import (e.g., encoders = generate(...))	caller_name.is_empty() in calls

Module-level calls are critical. For each call with empty caller_name:

1. Find the callee in the genome (same-file lookup, then suffix match)
2. Mark it as entrypoint
3. These are functions called during import — pruning them breaks the module

for call in &combined.calls {
    if !call.caller_name.is_empty() { continue; }
    // Find callee and mark as entrypoint
    if let Some(idx) = genome.find_node(&caller_file, &call.callee_name) {
        genome.mark_entrypoint(idx);
    }
}

Phase 5: Safety Filters

After classify_all(), collect unreachable candidates. Apply these filters before pruning:

Filter 1: Library-only — Only prune functions in the library source, never in tests.

Filter 2: No __init__.py — Never prune public API surface files.

Filter 3: No dunder methods — __init__, __repr__, etc. are protocol methods.

Check the BARE name (after stripping class prefix). This catches both class-bound dunders (ClassName.__init__) AND module-level PEP 562 dunders like __getattr__/__dir__ that have no class prefix.

let bare_name = name.rsplit('.').next().unwrap_or(name);
if bare_name.starts_with("__") && bare_name.ends_with("__") {
    continue;
}

Empirical bug found 2026-04-17: filtering name.contains("__") && name.contains('.') missed PEP 562 module-level __getattr__ in Click and Starlette → 5 wrongly-pruned dunders + RuntimeErrors. Always check the bare name pattern, not the fully-qualified form.

Filter 4: Value-reference scanning — THE MOST IMPORTANT FILTER.

Functions passed as values (callbacks, default args, variable assignments) are invisible to call-site analysis. Detect them with word-boundary text scanning:

// Load all library source into one string
let all_source = read_all_py_files(src_dir);

for candidate in unreachable_functions {
    let bare_name = candidate.name.rsplit('.').next().unwrap();

    // Count word-boundary occurrences (not substring!)
    let word_refs = count_word_references(&all_source, bare_name);
    let def_count = all_source.matches(&format!("def {bare_name}")).count();

    // If name appears more times than it's defined, it's referenced as a value
    if word_refs > def_count {
        skip; // Don't prune — it's used as callback/default/assignment
    }
}

Word-boundary matching is essential. Substring matching causes false positives: "get" matches inside "target", "budget", "widget". Check that characters before/after the match are NOT alphanumeric or underscore.

fn count_word_references(text: &str, word: &str) -> usize {
    let mut count = 0;
    let wb = word.as_bytes();
    let tb = text.as_bytes();
    for i in 0..=(tb.len() - wb.len()) {
        if &tb[i..i + wb.len()] == wb {
            let before_ok = i == 0
                || (!tb[i-1].is_ascii_alphanumeric() && tb[i-1] != b'_');
            let after_ok = i + wb.len() >= tb.len()
                || (!tb[i + wb.len()].is_ascii_alphanumeric() && tb[i + wb.len()] != b'_');
            if before_ok && after_ok { count += 1; }
        }
    }
    count
}

Phase 6: Prune and Retest

6a. Copy the project — Never modify the original.

cp -r $PROJECT_DIR /tmp/scalpel-test-pruned

6b. Stub unreachable functions using scalpel_prune::stub_py_functions:

for (file, funcs) in &prunable_by_file {
    let specs: Vec<(&str, u32, u32)> = funcs.iter()
        .map(|(name, ls, le)| (name.rsplit('.').next().unwrap(), *ls, *le))
        .collect();
    scalpel_prune::stub_py_functions(&target_path, &specs, "project", false)?;
}

Stubbing replaces function bodies with raise RuntimeError("scalpel: function 'X' was pruned..."). This is safer than deletion — if a pruned function IS called, you get a clear error message instead of NameError.

6c. Install pruned version and run tests:

/tmp/scalpel-test-venv/bin/pip install -e /tmp/scalpel-test-pruned --quiet --no-deps
cd /tmp/scalpel-test-pruned
/tmp/scalpel-test-venv/bin/python -m pytest tests/ -q --timeout=60 -p no:sugar

6d. Restore original:

/tmp/scalpel-test-venv/bin/pip install -e $PROJECT_DIR --quiet --no-deps

Phase 7: Report

Report with these exact numbers:

=== Prune Safety Test Results ===
Project: <name> (<url or path>)
Python: <version>

Analysis:
  Functions extracted: <N> (library: <N>, tests: <N>)
  Call sites: <N>
  Genome: <N> nodes, <N> edges

Classification:
  Reachable: <N>
  Unreachable: <N>
  Entrypoints marked: <N> (+ <N> module-level targets)

Safety Filters:
  Skipped (value-referenced): <N>
  Skipped (__init__/dunder): <N>
  Final prunable: <N> in <N> files

Pruning:
  Functions stubbed: <N>
  Files modified: <N>

Test Results:
  Baseline: <N> passed
  After pruning: <N> passed (diff: <N>)
  Scalpel RuntimeErrors: <N>

Verdict: SAFE / UNSAFE

Verdict rules:

• SAFE: diff <= 2 AND zero scalpel RuntimeErrors
• UNSAFE: diff > 2 OR any scalpel RuntimeError in test output

If UNSAFE, list each failing test and the pruned function that caused it. This is a real bug in Scalpel's analysis that needs fixing.

Known Limitations (Static-Only Analysis)

These patterns cause false positives that the safety filters catch:

Pattern	Example	Why Missed	Filter
Function as default arg	Default(generate_unique_id)	Not a call site	Value-reference scan
Callback/strategy	tg.start_soon(handler)	Argument, not callee	Value-reference scan
Instance method dispatch	app.get("/path")	Can't resolve app type	Class method entrypoint
Module-level side effect	cache = build_cache()	Empty caller_name	Module-level entrypoint
Decorator magic	@app.route(...)	Framework dispatch	Class method entrypoint
__all__ exports	__all__ = ["func1", "func2"]	String reference	__init__ exclusion

With runtime tracing (V8 coverage, Python sys.monitoring), these patterns are all detected correctly. Static-only analysis requires the safety filters.

Red Flags

Situation	Action
Baseline tests fail	STOP. Fix the project setup first.
>50% of functions classified unreachable	Entrypoint marking is wrong. Review Phase 4 rules.
Any scalpel RuntimeError in pruned tests	False positive found. Don't commit — investigate root cause.
Prunable count is 0	Normal for well-tested projects with static-only analysis. Pipeline is validated even if nothing is pruned.
Test count diff > 2	Check if diff is from pruning or from running tests in a temp dir (path comparison issues).

Cleanup

After the test, clean up:

rm -rf /tmp/scalpel-test-project /tmp/scalpel-test-pruned /tmp/scalpel-test-venv
rm -f tests/_prune_safety_test.rs  # remove temporary test file