bottleneck-triage

name	bottleneck-triage
description	LOAD when flawed crashes, hangs, OOMs, produces 0 findings, returns 0-byte output files, takes >2 min on a single rule, or exhibits any unexpected behavior during real-world repo analysis. Also load when investigating WHY a previous scan failed, when profiling performance, or when an agent reports a blocker from SSH evaluation.

Diagnose why flawed fails or underperforms on a real-world repository. This skill encodes the diagnostic methodology evolved through multiple real-world debugging sessions — every step is backed by a specific production incident where this exact approach resolved the issue.

Step 1: Read the Symptoms Before Touching Anything

Before running any commands, gather what you already have:

Check	Where to look
Exit code	The command output or evaluation log
stderr output	*.stderr files in results directory
JSON output size	ls -la *.json — 0 bytes is ALWAYS a bug signal
Timing	File timestamps (ls -la --time-style=long-iso), evaluation.log, or -v output
Phase that completed last	-v output shows [L1:semgrep] completed in Xs — missing completion = stall point
Gaps in findings	jq '.findings[0].gaps' output.json — engine self-reports limitations

The single most common mistake: re-running the same command instead of reading what the FIRST run already told you. Stop. Read. Think.

Step 2: Classify the Symptom

Exit code	Meaning	Diagnostic action	Historical example
137	OOM (SIGKILL)	Measure RSS per phase with /usr/bin/time -v	DISC-046: reexport resolution infinite string growth → 15GB RSS on 146-file repo
124	timeout utility killed it	Check which phase was active when killed — -v output	CTFd: L2 conversion modules had uncached O(n) scans → 10 min/tier
143	SIGTERM from timeout	Same as 124 but via SIGTERM not SIGKILL	Same CTFd issue — the corpus eval script wasn't checking exit 143
0 + 0 findings	Engine ran but found nothing	Check: did --semantic run? Were providers activated? Run flawed providers list against the repo	L2/L3/L4 "0 findings" on flaskbb was actually wrong --rules-dir paths
0 + 0-byte JSON	Killed between computation and output flush	This means --json writes atomically at end — any kill = 0 bytes	CTFd: every tier killed by timeout → every JSON file 0 bytes
Python traceback	Unhandled exception	Read the traceback — file:line tells you the layer	Usually missing conversion path in L2 or type mismatch at layer boundary
0 + findings but all FP	Engine works but analysis is shallow	Load the finding-audit skill instead	DISC-047: 94% FP rate from one missing L1 extractor

Step 3: Isolate — Run Each Phase Independently

Do NOT run the full pipeline repeatedly. Isolate:

# Phase A: Does L1 index complete?
timeout 300 flawed index /path/to/repo -v 2>&1 | tee /tmp/index.log
# Look for: "[L1:total] completed in Xs"
# If missing: L1 is the blocker. Check RSS, check resolution passes.

# Phase B: Does a single trivial L0 rule work? (no semantic needed)
timeout 60 flawed scan --no-semantic -i g001-multi-container-read /path/to/repo -v 2>&1

# Phase C: Does L2 semantic build work? (semantic, single rule)
timeout 120 flawed scan -i g001-multi-container-read /path/to/repo -v 2>&1
# The difference between B and C timing IS the L2 cost.

# Phase D: Escalate one tier at a time — STOP at first failure
for tier in L0_gadgets L1_invariants L2_targeted L3_comparative L4_ultra_specific; do
    echo "=== $tier ==="
    timeout 120 flawed scan --rules-dir "src/flawed/_rules/$tier/" /path/to/repo --json 2>&1 | head -5
    echo "exit=$?"
    # If this tier fails, do NOT continue to the next
done

Step 4: Profile the Specific Bottleneck

Once you know WHICH phase stalls:

L1 extraction bottleneck (rare after OOM fix):

# Measure per-phase RSS
/usr/bin/time -v flawed index /path/to/repo -vv 2>&1 | grep -E 'Maximum resident|\[L1:'
# The -vv flag shows per-file extraction times for the 10 slowest files

L2 semantic bottleneck (most common):

# Profile L2 build
python -c "
import cProfile, pstats
from flawed._index._structural import extract_structural
from flawed._semantic._matching import WebApp
from pathlib import Path

repo = Path('/path/to/repo')
idx = extract_structural(repo)
cProfile.run('WebApp.from_index(idx, repo)', '/tmp/l2_profile.prof')
pstats.Stats('/tmp/l2_profile.prof').sort_stats('cumulative').print_stats(20)
"
# WRITE THIS AS A LOCAL SCRIPT, transfer via scp if running on sandbox

Rule execution bottleneck (check per-rule timing):

# Run one rule at a time to find the slow one
for rule in src/flawed/_rules/L0_gadgets/*.py; do
    id="$(basename "$rule" .py)"
    echo -n "$id: "
    timeout 30 /usr/bin/time -f "%e seconds" flawed scan -i "$id" /path/to/repo 2>&1 | tail -1
done

Step 5: Common Root Causes — Lookup Table

Symptom pattern	Root cause category	Where to look	Fix pattern
RSS grows linearly without bound	Infinite loop / unbounded data structure	Resolution passes, rewrite loops	Add growth guard + filter at source
O(n) operation called N times = O(n²)	Missing cache/index	Any for x in all_items inside another loop	Build dict index, compute once
Works on small repo, fails on large	Algorithmic complexity	Same code path, larger input triggers	Profile with cProfile, find the hot loop
0-byte JSON after timeout	Output written only at completion	pipeline.py JSON emission	Add streaming/incremental output OR per-layer output
"No detection rules matched"	Path mismatch	Check flawed rules output	Verify --rules-dir exists and contains .py files
All findings are FP	Missing extractor or provider	L1 (decorators, class attrs) or L2 (provider patterns)	Load finding-audit skill, identify the dominant gap
Specific framework not detected	Missing or broken provider	src/flawed/_semantic/providers/	Check provider exists, check conversion path is wired

Step 6: The Fix — Always Systemic, Never Band-Aid

When you find the root cause:

1. Identify the bug CLASS, not just the instance. The flaskbb OOM wasn't "self-referential reexports" — it was "any rewrite rule that causes string growth." Test mutual cycles, transitive chains, deep valid chains.

2. Fix at TWO levels:

   • Source filter: prevent bad data from entering (_build_reexport_map rejecting self-embedding entries)
   • Defense-in-depth: catch it if filtering fails (growth guard in _resolve_reexported_fqn)

3. Write tests for the wider class FIRST (TDD):

   • Test the exact failure case (must fail before fix, pass after)
   • Test 3-5 related variants (mutual cycles, transitive, deep valid chains)
   • Test that the fix doesn't break legitimate behavior

4. File via planq create engine-v1 -t "DISC: <title>" -d "<description>" -p <priority>.

Step 7: Anti-Patterns — Things That Waste Hours

Anti-pattern	Why it's wrong	Do this instead
Re-running same command	Definition of insanity	Read the output from the first run
Raising timeout from 5m to 30m	Hides the bug, wastes 30m	Profile WHY it takes >5m
Running L3+ after L2 fails	L3 depends on L2 — cascading waste	Fix L2 first
Ad-hoc Python over SSH	Fragile, unreviewable, untestable	Write script locally in scripts/, transfer via scp
timeout 3600 flawed scan ...	1-hour timeout ≈ no timeout	Use 120s for known-good, 300s maximum for investigation
Assuming 0 findings = working	0 findings + 0-byte output = BROKEN	Always check file sizes and exit codes
Fixing without tests	It'll break again	Write the failing test first, verify it fails, then fix