| name | deep-crawl |
| description | Exhaustive LLM-powered codebase investigation for optimal AI agent onboarding. Builds on X-Ray signals with unlimited investigation budget to produce the highest-quality onboarding document possible. |
Deep Crawl
Systematic codebase investigation producing comprehensive onboarding
documents optimized for AI agent consumption via CLAUDE.md delivery with
prompt caching. Depth over brevity — include everything that saves a
downstream agent from opening files.
Core metric: File-reads saved per onboarding token. Every token in your output should reduce the number of files a downstream agent needs to open before it can confidently make changes.
When to Use
- Use
/deep-crawl full when generation cost is not a constraint and you want
the highest-quality onboarding document for many future agent sessions
- Use
@deep_crawl full as a sequential fallback if /deep-crawl is unavailable
- Use repo_xray for quick interactive analysis where cost matters
- Always run xray first — deep crawl requires xray output as input
Commands
| Command | Mode | What It Does |
|---|
/deep-crawl full | Orchestrated (parallel sub-agents) | Plan + parallel investigation + assemble + cross-reference + validate + deliver |
@deep_crawl full | Sequential (single-agent fallback) | Same pipeline, sequential investigation |
@deep_crawl plan | Sequential | Generate investigation plan only |
@deep_crawl resume | Sequential | Continue from last checkpoint |
@deep_crawl validate | Sequential | QA an existing DEEP_ONBOARD.md |
@deep_crawl refresh | Sequential | Update for code changes |
@deep_crawl focus ./path | Sequential | Deep crawl a specific subsystem |
/deep-crawl full <github-url> | Orchestrated | Clone remote repo, auto-run xray, then full crawl pipeline |
Remote Repository Support
When a GitHub URL (or gh:owner/repo shorthand) is passed as an argument to /deep-crawl full,
the skill clones the repository to a local temp directory and runs the full pipeline against it.
Supported URL formats:
https://github.com/owner/repohttps://github.com/owner/repo.gitgh:owner/repogit@github.com:owner/repo.git
Key differences from local crawl:
- Repository is cloned with full history (for git log/blame) to
.deep_crawl/repo/
- Xray is run automatically — no manual prerequisite step
DEEP_CRAWL_ROOT points to the clone directory instead of $(pwd)- Sub-agent
{ROOT_PATH} is set to the clone directory
- Phase 6 delivers output to
output/{repo-name}/ (same structure as local crawls)
- CLAUDE.md in the clone is not modified (read-only analysis of external repo)
- Cleanup is manual:
rm -rf .deep_crawl/repo/ when done
What stays the same: All six phases, quality gates, investigation protocols,
evidence standards, and validation — the entire pipeline is identical.
Evidence Standards
| Tag | Standard | Example |
|---|
| [FACT] | Read specific code, cite file:line | "3x retry with backoff [FACT] (payments.ts:89 / stripe.py:89)" |
| [PATTERN] | Observed in >=3 examples, state count | "DI via init [PATTERN: 12/14 services]" |
| [ABSENCE] | Searched and confirmed non-existence | "No rate limiting [ABSENCE: grep — 0 hits]" |
No inferences or unverified signals in the output document.
Citation density floor: Assembled sections have tiered density requirements:
high-evidence sections (impact, gotchas, contracts) >= 3.0, medium (module index,
interfaces, playbooks, error handling) >= 2.0, narrative (critical paths, conventions) >= 1.0,
structural (glossary, reading order) >= 0. Playbooks individually >= 3.0 per 100 words.
Investigation findings (raw) should target >= 5.0 per 100 words.
Output
| File | Purpose | Location |
|---|
| DEEP_ONBOARD.md | Onboarding document (unrestricted, value-driven) | output/{repo-name}/ |
| xray.md | Structural analysis companion | output/{repo-name}/ |
| CLAUDE.md update | Auto-delivery to all sessions (local only) | project root |
| crawl.log | Full pipeline debug log (gates, spawns, re-spawns, stats) | output/{repo-name}/data/ |
| All intermediate data | Findings, sections, plans, validation | output/{repo-name}/data/ |
Prerequisites
Before starting, verify:
- X-Ray output exists at
output/$REPO_NAME/data/xray.json and output/$REPO_NAME/xray.md
- If missing, tell the user: "Run
python xray.py . --output both first."
Six-Phase Workflow
Phase 0: SETUP (Context Management)
Goal: Establish working environment and context management strategy.
All intermediate state lives on disk, not in conversation context.
# === REMOTE REPOSITORY DETECTION ===
# If the user passed a GitHub URL, clone it and set DEEP_CRAWL_ROOT.
# Otherwise, DEEP_CRAWL_ROOT defaults to the current working directory.
DEEP_CRAWL_REMOTE="${1:-}" # first argument, if any
DEEP_CRAWL_ROOT="$(pwd)"
if [[ "$DEEP_CRAWL_REMOTE" =~ ^(https://github\.com/|git@github\.com:|gh:) ]]; then
REPO_DIR=".deep_crawl/repo"
# Clean previous clone
[ -d "$REPO_DIR" ] && rm -rf "$REPO_DIR"
# Normalize gh:owner/repo shorthand to full URL
if [[ "$DEEP_CRAWL_REMOTE" == gh:* ]]; then
DEEP_CRAWL_REMOTE="https://github.com/${DEEP_CRAWL_REMOTE#gh:}"
fi
echo "Cloning remote repository: $DEEP_CRAWL_REMOTE"
gh repo clone "$DEEP_CRAWL_REMOTE" "$REPO_DIR" 2>/dev/null \
|| git clone "$DEEP_CRAWL_REMOTE" "$REPO_DIR"
if [ ! -d "$REPO_DIR/.git" ]; then
echo "HALT: Clone failed. Check the URL and your access permissions."
exit 1
fi
DEEP_CRAWL_ROOT="$REPO_DIR"
echo "Remote repo cloned to: $DEEP_CRAWL_ROOT"
echo "DEEP_CRAWL_MODE=remote"
# Auto-run xray on cloned repo
# Locate xray.py: check skill source directory, then cwd, then PATH
XRAY_PY=""
SKILL_SOURCE=$(readlink -f ~/.claude/skills/deep-crawl 2>/dev/null || echo "")
if [ -n "$SKILL_SOURCE" ] && [ -f "$(dirname "$SKILL_SOURCE")/../../xray.py" ]; then
XRAY_PY="$(dirname "$SKILL_SOURCE")/../../xray.py"
elif [ -f "xray.py" ]; then
XRAY_PY="./xray.py"
elif command -v xray.py >/dev/null 2>&1; then
XRAY_PY="xray.py"
fi
if [ -n "$XRAY_PY" ]; then
echo "Running xray: python $XRAY_PY $DEEP_CRAWL_ROOT --output both --repo-name $REPO_NAME"
python "$XRAY_PY" "$DEEP_CRAWL_ROOT" --output both --repo-name "$REPO_NAME"
else
echo "HALT: Cannot find xray.py. Provide its path or run manually:"
echo " python /path/to/xray.py $DEEP_CRAWL_ROOT --output both"
fi
else
echo "DEEP_CRAWL_MODE=local"
fi
# Determine repo name and output directory
REPO_NAME=$(cd "$DEEP_CRAWL_ROOT" && git remote get-url origin 2>/dev/null | sed 's|.*/||; s|\.git$||' || basename "$DEEP_CRAWL_ROOT")
OUTPUT_DIR="output/$REPO_NAME"
echo "REPO_NAME=$REPO_NAME"
echo "OUTPUT_DIR=$OUTPUT_DIR"
# Create working directory structure
mkdir -p .deep_crawl/findings/{traces,modules,cross_cutting,conventions,impact,playbooks,calibration} \
.deep_crawl/batch_status \
.deep_crawl/sections \
.deep_crawl/agent_logs/prompts \
.deep_crawl/agent_logs/results \
"$OUTPUT_DIR/data"
# Initialize crawl log
CRAWL_LOG="$OUTPUT_DIR/data/crawl.log"
echo "=== Deep Crawl Log ===" > "$CRAWL_LOG"
echo "Started: $(date -Iseconds)" >> "$CRAWL_LOG"
echo "Repo: $REPO_NAME" >> "$CRAWL_LOG"
echo "Root: $DEEP_CRAWL_ROOT" >> "$CRAWL_LOG"
echo "Mode: ${DEEP_CRAWL_MODE:-local}" >> "$CRAWL_LOG"
echo "" >> "$CRAWL_LOG"
# Initialize orchestrator event log
ORCH_LOG=".deep_crawl/agent_logs/orchestrator.log"
FILE_COUNT=$(find "$DEEP_CRAWL_ROOT" -type f -not -path "*/.git/*" -not -path "*/node_modules/*" -not -path "*/dist/*" 2>/dev/null | wc -l)
echo "$(date -Iseconds) PHASE 0 SETUP repo=$REPO_NAME files=$FILE_COUNT" > "$ORCH_LOG"
# Verify xray output exists
test -f "$OUTPUT_DIR/data/xray.json" && echo "READY" || echo "Run: python xray.py $DEEP_CRAWL_ROOT --output both"
# Clean stale data from previous crawl
if [ -f .deep_crawl/CRAWL_PLAN.md ]; then
PREV_HASH=$(head -1 .deep_crawl/CRAWL_PLAN.md | grep -oP '[a-f0-9]{7,}' || echo "unknown")
echo "⚠️ PREVIOUS CRAWL FOUND (hash: $PREV_HASH)"
echo "Cleaning stale data for fresh crawl..."
rm -rf .deep_crawl/findings/* .deep_crawl/batch_status/* .deep_crawl/sections/*
rm -f .deep_crawl/CRAWL_PLAN.md .deep_crawl/SYNTHESIS_INPUT.md
rm -f .deep_crawl/DRAFT_ONBOARD.md .deep_crawl/DEEP_ONBOARD.md
rm -f .deep_crawl/VALIDATION_REPORT.md .deep_crawl/REFINE_LOG.md
fi
# Check for existing crawl state (resumability)
if [ -f .deep_crawl/CRAWL_PLAN.md ]; then
echo "PREVIOUS CRAWL FOUND"
head -5 .deep_crawl/CRAWL_PLAN.md
git -C "$DEEP_CRAWL_ROOT" log --oneline -1
echo "If hashes match, run @deep_crawl resume"
echo "If not, this is a stale crawl — starting fresh"
fi
DEEP_CRAWL_ROOT usage: All subsequent phases use $DEEP_CRAWL_ROOT wherever the codebase
root is referenced. For local crawls this equals $(pwd) (backward compatible). For remote
crawls it points to .deep_crawl/repo/. Sub-agent prompts must set {ROOT_PATH} to
the value of $DEEP_CRAWL_ROOT.
Context management rules:
- Write findings to disk immediately after each investigation task
- Read only what's needed for the current task
- Batch related reads — trace an entire request path before moving on
- Before Phase 3 (ASSEMBLE), concatenate all findings and read fresh
- If context gets full, checkpoint and suggest
@deep_crawl resume
- Sub-agents write findings directly to disk; you never read findings content until Phase 3
- Between batches, check only batch_status/ sentinel files, not findings content
- In Phase 3, assembly sub-agents read findings and write section files to .deep_crawl/sections/.
The orchestrator reads only sentinel files until assembly. Gotchas extracted via bash pre-step.
Agent logging protocol (apply at every spawn point):
Before spawning any sub-agent:
- Write the exact prompt text to
.deep_crawl/agent_logs/prompts/{agent_id}.md
- Append to orchestrator.log:
echo "$(date -Iseconds) SPAWN {agent_id} prompt=agent_logs/prompts/{agent_id}.md" >> "$ORCH_LOG"
After agent completion:
- Write the Agent tool's return text + output file stats to
.deep_crawl/agent_logs/results/{agent_id}.md
- Append to orchestrator.log:
echo "$(date -Iseconds) DONE {agent_id} files={count} words={total} facts={total}" >> "$ORCH_LOG"
After any quality gate:
- Append to orchestrator.log:
echo "$(date -Iseconds) GATE {agent_id} {PASS|FAIL} {metrics}" >> "$ORCH_LOG"
After any re-spawn:
- Append to orchestrator.log:
echo "$(date -Iseconds) RESPAWN {agent_id} reason=\"{reason}\"" >> "$ORCH_LOG"
Agent IDs follow this naming convention:
- Calibration:
cal_a, cal_b, cal_c
- Investigation: task ID from CRAWL_PLAN.md (e.g.,
P1.1, P2.3, P4.2)
- Assembly:
S1, S2, S3a, S3b, S4, S5, S6
- Cross-reference:
phase4_crossref
- Validation:
phase5_validate
- Gap investigation:
gap_{name}
Phase 0b: PRE-FLIGHT DIAGNOSTICS
Goal: Catch stale inputs and repo characteristics before investing tokens in investigation.
# === PRE-FLIGHT DIAGNOSTICS ===
echo "=== Pre-Flight ==="
# Xray freshness — robust hash comparison
XRAY_HASH=$(python3 -c "
import json
d = json.load(open('$OUTPUT_DIR/data/xray.json'))
print(d.get('git_commit', d.get('commit_hash', '?')))
" 2>/dev/null)
HEAD=$(git -C "$DEEP_CRAWL_ROOT" rev-parse HEAD 2>/dev/null || echo "no-git")
if [ "$XRAY_HASH" != "?" ] && [ "$HEAD" != "no-git" ]; then
# Prefix match handles short-vs-full hash
if [[ ! "$HEAD" == "$XRAY_HASH"* ]] && [[ ! "$XRAY_HASH" == "$HEAD"* ]]; then
echo "HALT: Xray is stale (xray: ${XRAY_HASH:0:7}, HEAD: ${HEAD:0:7}). Re-run xray."
else
echo "Xray matches HEAD: ${HEAD:0:7}"
fi
else
[ "$XRAY_HASH" = "?" ] && echo "WARNING: Xray has no git_commit field. Proceeding without freshness check."
fi
# Repo characteristics — detect language
PY_COUNT=$(find "$DEEP_CRAWL_ROOT" -name "*.py" -not -path "*/.git/*" -not -path "*/node_modules/*" 2>/dev/null | wc -l)
TS_COUNT=$(find "$DEEP_CRAWL_ROOT" \( -name "*.ts" -o -name "*.tsx" -o -name "*.js" -o -name "*.jsx" \) -not -path "*/.git/*" -not -path "*/node_modules/*" -not -path "*/dist/*" 2>/dev/null | wc -l)
if [ "$TS_COUNT" -gt "$PY_COUNT" ]; then
CODE_LANG="typescript"
FILE_COUNT=$TS_COUNT
TEST_COUNT=$(find "$DEEP_CRAWL_ROOT" \( -name "*.test.ts" -o -name "*.spec.ts" -o -name "*.test.js" -o -name "*.test.tsx" \) -not -path "*/node_modules/*" 2>/dev/null | wc -l)
echo "Language: TypeScript/JavaScript | Files: $FILE_COUNT | Test files: $TEST_COUNT"
else
CODE_LANG="python"
FILE_COUNT=$PY_COUNT
TEST_COUNT=$(find "$DEEP_CRAWL_ROOT" -name "test_*.py" -o -name "*_test.py" | wc -l)
echo "Language: Python | Files: $FILE_COUNT | Test files: $TEST_COUNT"
fi
echo "CODE_LANG=$CODE_LANG"
[ "$FILE_COUNT" -gt 5000 ] && echo "WARNING: Very large repo. Consider focused crawl."
[ "$TEST_COUNT" -eq 0 ] && echo "WARNING: No test files detected. Testing section may be thin. (Note: in-source testing via import.meta.vitest won't be detected by file naming.)"
# Framework detection
if [ "$CODE_LANG" = "typescript" ]; then
for fw in express next nestjs fastify koa hapi commander yargs oclif react vue angular jest vitest mocha prisma typeorm sequelize mongoose graphql trpc zod valibot; do
grep -rl "\"$fw\"" --include="*.json" "$DEEP_CRAWL_ROOT" 2>/dev/null | grep -v node_modules | head -1 >/dev/null 2>&1 && echo "Framework: $fw"
done
else
for fw in fastapi flask django click typer torch tensorflow airflow dagster numpy scipy asyncio aiohttp paramiko boto3 pluggy stevedore; do
grep -rl "$fw" --include="*.py" "$DEEP_CRAWL_ROOT" 2>/dev/null | head -1 >/dev/null 2>&1 && echo "Framework: $fw"
done
fi
Halt condition: If xray hash doesn't match HEAD, stop and tell user to re-run xray. Other warnings are logged to .deep_crawl/PREFLIGHT.md for context during investigation planning.
Save all pre-flight output to .deep_crawl/PREFLIGHT.md for Phase 1 consumption. Also append pre-flight output to $CRAWL_LOG under a ## Pre-flight header.
Phase 1: PLAN (Build the Crawl Agenda)
echo "## Phase 1: PLAN — $(date -Iseconds)" >> "$CRAWL_LOG"
echo "$(date -Iseconds) PHASE 1 PLAN" >> "$ORCH_LOG"
Input: X-Ray JSON output including investigation_targets.
- Read
output/$REPO_NAME/xray.md for orientation
- Read
investigation_targets from output/$REPO_NAME/data/xray.json
- Read
git.function_churn and git.velocity from output/$REPO_NAME/data/xray.json. Files with accelerating velocity or high function-level churn should be prioritized for investigation.
- Detect all applicable domain facets using indicators from
.claude/skills/deep-crawl/configs/domain_profiles.json. A repo can match multiple facets (e.g., a Django app with Celery workers and a CLI gets web_api + async_service + cli_tool). Union their additional_investigation tasks into the crawl plan. If no facet matches, use library as default. Read .deep_crawl/PREFLIGHT.md for framework detection results to guide facet matching.
For each matched facet:
- Add its
additional_investigation tasks to the crawl plan at P4 priority (cross-cutting concerns)
- Add its
additional_output_sections to the assembly plan for the appropriate sub-agent (S3 or S4)
- Add its
grep_patterns to Protocol C investigation prompts for related cross-cutting agents
- Record all matched facets in CRAWL_PLAN.md header:
Domain Facets: web_api, async_service, cli_tool
If facet investigation tasks push any batch beyond its max agent limit, split into sub-batches.
Use the facet's primary_entity to determine the adversarial simulation scenario in Phase 5.
If multiple facets match, the adversarial simulation should use the primary facet's entity.
4. Produce a prioritized crawl plan using the template at .claude/skills/deep-crawl/templates/CRAWL_PLAN.md.template
5. Save to .deep_crawl/CRAWL_PLAN.md
6. Module coverage pre-check (mandatory). After saving the plan, verify P2+P3 task count meets the coverage target:
# === MODULE COVERAGE PRE-CHECK ===
P23_COUNT=$(grep -c '^\- \[ \] P[23]\.' .deep_crawl/CRAWL_PLAN.md 2>/dev/null || echo 0)
TARGET=$(python3 -c "
import json
d = json.load(open('$OUTPUT_DIR/data/xray.json'))
file_count = len(d.get('files', d.get('file_list', [])))
print(max(10, file_count // 40))
" 2>/dev/null)
echo "P2+P3 tasks: $P23_COUNT, coverage target: $TARGET"
TOTAL_TASKS=$(grep -c '^\- \[ \]' .deep_crawl/CRAWL_PLAN.md 2>/dev/null || echo 0)
FACETS=$(head -20 .deep_crawl/CRAWL_PLAN.md | grep -oP 'Domain Facets: .*' | sed 's/Domain Facets: //' || echo "none")
echo "$(date -Iseconds) PHASE 1 PLAN tasks=$TOTAL_TASKS facets=$FACETS" >> "$ORCH_LOG"
If P23_COUNT < TARGET:
a. Read xray.json import graph: identify all modules with imported_by count >= 3 that are NOT already P2 or P3 tasks
b. Sort by imported_by count descending
c. Add the top (TARGET - P23_COUNT) modules as new P3 tasks to CRAWL_PLAN.md
d. Hop promotion: Any module that appears as an intermediate hop in a P1 trace task description AND does not already have a P2/P3 task gets added as a P3 task (these are modules the traces pass through — they deserve deep-reads)
e. Update the Progress table's P3 total
f. Log: Coverage pre-check: {P23_COUNT} tasks → {new_count} (added {added} modules, target {TARGET})
g. Barrel file analysis (TypeScript only): If CODE_LANG is typescript, identify index.ts files that are mostly re-exports (high export count, low logic). These are architectural chokepoints — if they create circular imports, many modules break. Add the top 3 barrel files as P3 tasks if not already present.
- TypeScript-specific prioritization signals: If CODE_LANG is typescript:
- Modules with high
as any / @ts-ignore density should be prioritized (type safety gaps indicate under-documented behavior)
- Files listed in
ts_specific.module_augmentations are coupling risks — add as P3 tasks
- Check for
ts_specific.any_density in xray.json — if explicit_any + as_any_assertions > 20, add a P4 cross-cutting task: "Audit type safety escape hatches and document which are intentional vs technical debt"
Prioritization logic (by information density):
| Priority | Task Type | Rationale |
|---|
| 1 | Request traces | Highest file-reads-saved per output token |
| 2 | High-uncertainty module deep reads | Name and signature tell you nothing |
| 3 | Pillar behavioral summaries | Most-depended-on modules |
| 4 | Cross-cutting concerns | Learn once, apply everywhere |
| 5 | Conventions and patterns | Prevents style violations |
| 6 | Gap investigation | Catches what xray missed |
Within each priority level, order tasks by information density: request traces by estimated hop count descending (longer traces reveal more cross-module behavior). Module deep reads by uncertainty score descending (highest-uncertainty modules first).
Use extended thinking here to reason about investigation priorities for this specific codebase.
Phase 1b: CALIBRATE (Repo-Specific Exemplar Discovery)
Goal: Before bulk investigation, produce repo-specific quality exemplars by investigating a small number of high-value targets at elevated depth. Their output becomes the quality reference for all Phase 2 sub-agents, replacing static exemplars.
Scope by repo size (from Phase 0b FILE_COUNT):
| Files | Calibration |
|---|
| <= 10 | Skip — use structural templates only |
| 11-30 | 1 target (CAL_B only) |
| >= 31 | Full 3 targets |
Target selection (deterministic, no LLM judgment):
- CAL_A (Protocol A trace): First P1 task from CRAWL_PLAN.md (longest trace)
- CAL_B (Protocol B module): First P2 task (highest uncertainty module), or first P3 if no P2
- CAL_C (Protocol C cross-cutting): First P4 task (error handling — universal)
Elevated quality floor (calibration must exceed normal investigation floor):
- 400 words (normal: 200), 10 [FACT] (normal: 5), density 6.0/100w (normal: 5.0)
- For traces: follow EVERY branch. For modules: document EVERY public method
- For cross-cutting: read >= 5 representative examples
Thresholds are defined in .claude/skills/deep-crawl/configs/quality_gates.json under calibration_findings.
Procedure:
echo "$(date -Iseconds) PHASE 1b CALIBRATE" >> "$ORCH_LOG"
- Parse CRAWL_PLAN.md and select targets per the rules above.
- Spawn calibration agents (1-3 depending on repo size). Each agent prompt follows the standard Phase 2 format with these modifications:
- Quality floor references
calibration_findings thresholds (400w, 10 FACT, 6.0/100w)
- Format guidance references
.claude/skills/deep-crawl/configs/exemplar_templates.md only (calibration findings don't exist yet)
- Output path:
.deep_crawl/findings/calibration/cal_{a|b|c}.md
- Sentinel:
touch .deep_crawl/batch_status/cal_{a|b|c}.done
- Before spawning each calibration agent, apply the agent logging protocol: write the prompt to
.deep_crawl/agent_logs/prompts/cal_{a|b|c}.md and log the SPAWN event.
- Wait for all calibration sentinels. After each calibration agent completes, write the Agent tool's return text to
.deep_crawl/agent_logs/results/cal_{a|b|c}.md and log the DONE event.
- Run calibration quality gate:
# === CALIBRATION QUALITY GATE ===
CAL_GATE=true
for f in .deep_crawl/findings/calibration/cal_*.md; do
[ -f "$f" ] || continue
WORDS=$(wc -w < "$f")
FACTS=$(grep -c '\[FACT' "$f" 2>/dev/null || echo 0)
[ "$WORDS" -eq 0 ] && continue
DENSITY=$((FACTS * 100 / WORDS))
if [ "$WORDS" -lt 400 ] || [ "$FACTS" -lt 10 ] || [ "$DENSITY" -lt 6 ]; then
echo "CAL FAIL: $(basename $f) — ${WORDS}w, ${FACTS} FACT, ${DENSITY}/100w density"
echo " (elevated floor: 400w, 10 FACT, 6.0/100w)"
CAL_GATE=false
fi
done
# Log calibration gate results
echo "## Calibration Gate: $([ "$CAL_GATE" = true ] && echo PASS || echo FAIL)" >> "$CRAWL_LOG"
for f in .deep_crawl/findings/calibration/cal_*.md; do
[ -f "$f" ] || continue
AGENT_ID=$(basename "$f" .md)
WORDS=$(wc -w < "$f"); FACTS=$(grep -c '\[FACT' "$f" 2>/dev/null || echo 0)
echo " $(basename $f): ${WORDS}w, ${FACTS} FACT" >> "$CRAWL_LOG"
[ "$WORDS" -eq 0 ] && continue
DENSITY=$((FACTS * 100 / WORDS))
if [ "$WORDS" -ge 400 ] && [ "$FACTS" -ge 10 ] && [ "$DENSITY" -ge 6 ]; then
echo "$(date -Iseconds) GATE $AGENT_ID PASS words=$WORDS facts=$FACTS density=${DENSITY}/100w" >> "$ORCH_LOG"
else
echo "$(date -Iseconds) GATE $AGENT_ID FAIL words=$WORDS facts=$FACTS density=${DENSITY}/100w" >> "$ORCH_LOG"
fi
done
echo "" >> "$CRAWL_LOG"
Failure handling:
- If calibration fails elevated gate: re-spawn once with corrective instructions
- If re-spawn passes normal gate (200w, 5 FACT) but not elevated: accept with warning
- If re-spawn fails normal gate: fall back to structural templates for that protocol
Integration:
Phase 2: CRAWL (Orchestrated Investigation)
echo "## Phase 2: CRAWL — $(date -Iseconds)" >> "$CRAWL_LOG"
echo "$(date -Iseconds) PHASE 2 CRAWL" >> "$ORCH_LOG"
Phase 2 uses parallel sub-agents to investigate the codebase. You act as the orchestrator — spawn investigation agents, monitor completion, and verify coverage. Do NOT perform investigation yourself (except as fallback).
Orchestration Procedure
Step 1: Parse and batch. Read CRAWL_PLAN.md. Group tasks into batches:
| Batch | Tasks | Protocol | Max Agents | Dependencies |
|---|
| 1 | All P1 (request traces) | A | 5 | None |
| 2 | All P2 + P3 (modules + pillars) | B | one per task | None |
| 3 | All P4 (cross-cutting concerns incl. async boundaries) | C | 6 | None |
| 4 | All P5 + P6 (conventions + gaps) | D + mixed | 4 | Batches 1-3 |
| 5 | P7 (change impact) + Coverage gaps | E + Mixed | 8 | Batches 1-4 |
| 6 | Change scenarios | F | 3 | Batches 1-5 |
Batches 1-3 are independent — launch them concurrently (all in a single message with multiple Agent tool calls). Batch 4 waits for 1-3 because convention detection and gap investigation benefit from earlier findings being on disk. Batch 5 (P7 impact + coverage gaps) waits for Batches 1-4 because impact analysis reads module findings. Batch 6 (change scenarios) waits for Batches 1-5 because playbooks reference impact findings.
If a batch has more tasks than its max agents, split into sequential sub-batches of max agents each. All sub-batches within a batch are independent.
Batch 2 unbatching rule: Spawn one agent per P2/P3 task — never batch multiple modules into one agent. Each module gets its own investigation context for richer, more distinct findings that map cleanly to individual ### subsections during assembly. If total P2+P3 tasks exceed 15, use sequential sub-batches of 15.
Batch 2 elevated quality floor: Because each unbatched agent investigates a single module with full context, findings must be deeper than the default floor. Batch 2 sub-agent prompts MUST use these thresholds instead of the default:
- Minimum 400 words (not 200)
- Minimum 10 [FACT] citations (not 5)
- Target density: >= 5.0 [FACT] per 100 words
The findings quality gate (Step 3b) MUST also check Batch 2 files against these elevated thresholds. Use the file path to identify Batch 2 findings: any file in
findings/modules/ (excluding 00_calibration.md) uses the elevated floor.
P7 relocation rationale: Change impact analysis (Protocol E) moved from Batch 2 to Batch 5 because impact analysis reads reverse dependency data enriched by module deep-reads. Running P7 after Batches 1-4 ensures impact agents have full module findings on disk.
Multi-facet batch adjustment: If domain facets added investigation tasks to P4,
Batch 3 may exceed its max of 6 agents. Split Batch 3 into sub-batches of 6 each.
Facet investigation tasks have no dependencies on Batch 1-2 and can run in any
Batch 3 sub-batch. The orchestrator must include the facet's grep_patterns in the
Protocol C prompt for facet-specific cross-cutting investigations.
Step 2: Spawn sub-agents. For each task in a batch, spawn a sub-agent using the Agent tool. Before spawning each agent, apply the agent logging protocol: write the prompt to .deep_crawl/agent_logs/prompts/{task_id}.md and log the SPAWN event to orchestrator.log. After each agent completes, write the return text to .deep_crawl/agent_logs/results/{task_id}.md and log the DONE event. Each sub-agent prompt must be self-contained with these sections:
You are investigating [CODEBASE] at [ROOT_PATH] for an onboarding document.
## Your Task
[Specific task from crawl plan — e.g., "Trace the primary CLI entry point from invocation to terminal side effect"]
## Investigation Protocol
[Full text of the relevant protocol (A, B, C, or D) copied verbatim from below]
## Evidence Standards
- [FACT]: Read specific code, cite file:line. Example: "retries 3x (payments.ts:89)" or "retries 3x (stripe.py:89)"
- [PATTERN]: Observed in >=3 examples, state count. Example: "DI via __init__ (12/14 services)"
- [ABSENCE]: Searched and confirmed non-existence. Example: "No rate limiting (grep — 0 hits)"
- Gotchas must be [FACT] claims with file:line.
- Never include inferences or unverified signals.
- ACTIVELY SEARCH for [ABSENCE] evidence. At each investigation step, ask: "what should exist here but doesn't?" Missing error handling, missing validation, missing tests, missing docs — these are high-value findings. Target: at least 1 [ABSENCE] per 5 [FACT] citations.
## Quality Floor (mechanically checked after you finish)
- Minimum 200 words
- Minimum 5 [FACT] citations
- Target density: >= 5.0 [FACT] per 100 words
If your file fails the check, you will be re-spawned to investigate deeper.
For format guidance, see .claude/skills/deep-crawl/configs/exemplar_templates.md.
For quality reference from this repo, see .deep_crawl/findings/calibration/cal_{type}.md.
## Output
Write findings to: [EXACT PATH — e.g., .deep_crawl/findings/traces/01_cli_run.md]
When done, write a sentinel: touch .deep_crawl/batch_status/[TASK_ID].done
## Constraints
- Read-only: never modify source code
- Do NOT spawn sub-agents yourself
- X-Ray output available at output/$REPO_NAME/data/xray.json and output/$REPO_NAME/xray.md for reference
Use run_in_background: true for all sub-agents within a batch to maximize parallelism. Note: If sub-agents run sequentially despite run_in_background: true, each sub-agent still gets a full context window for its investigation task, which is strictly better than sequential investigation in a single context.
Step 3: Monitor completion. After launching a batch, check for sentinel files:
ls .deep_crawl/batch_status/*.done 2>/dev/null | wc -l
When all expected sentinels exist, the batch is complete.
Step 3a: SNAPSHOT (before spawning batch). Record existing findings files so the quality gate only checks new ones. Log the batch spawn event:
# === FINDINGS SNAPSHOT (before spawning batch) ===
ls .deep_crawl/findings/{traces,modules,cross_cutting,conventions,impact,playbooks}/*.md \
2>/dev/null | sort > .deep_crawl/_pre_batch_files.txt
# Log batch spawn
BATCH_AGENT_COUNT=$(echo "$BATCH_TASKS" | wc -w) # set BATCH_TASKS to the task list for this batch
echo "## Batch $BATCH_NUM: Spawning $BATCH_AGENT_COUNT agents" >> "$CRAWL_LOG"
echo " Tasks: $BATCH_TASKS" >> "$CRAWL_LOG"
echo " Time: $(date -Iseconds)" >> "$CRAWL_LOG"
Step 3b: FINDINGS QUALITY GATE (batch-scoped, mandatory after each batch). After confirming all sentinel files exist, check only files produced in this batch:
# === FINDINGS QUALITY GATE (batch-scoped) ===
ls .deep_crawl/findings/{traces,modules,cross_cutting,conventions,impact,playbooks}/*.md \
2>/dev/null | sort > .deep_crawl/_post_batch_files.txt
GATE_PASS=true
while IFS= read -r f; do
WORDS=$(wc -w < "$f")
FACTS=$(grep -c '\[FACT' "$f" 2>/dev/null || echo 0)
if [ "$WORDS" -lt 200 ] || [ "$FACTS" -lt 5 ]; then
echo "FAIL: $(basename $f) — ${WORDS}w, ${FACTS} FACT (min: 200w, 5 FACT)"
GATE_PASS=false
fi
done < <(comm -13 .deep_crawl/_pre_batch_files.txt .deep_crawl/_post_batch_files.txt)
# Log batch completion and gate result
echo " Completed: $(date -Iseconds)" >> "$CRAWL_LOG"
echo "## Findings Gate (Batch $BATCH_NUM): $([ "$GATE_PASS" = true ] && echo PASS || echo FAIL)" >> "$CRAWL_LOG"
BATCH_PASSED=0; BATCH_TOTAL=0
while IFS= read -r f; do
AGENT_ID=$(basename "$f" .md)
WORDS=$(wc -w < "$f"); FACTS=$(grep -c '\[FACT' "$f" 2>/dev/null || echo 0)
echo " $(basename $f): ${WORDS}w, ${FACTS} FACT" >> "$CRAWL_LOG"
BATCH_TOTAL=$((BATCH_TOTAL + 1))
if [ "$WORDS" -ge 200 ] && [ "$FACTS" -ge 5 ]; then
echo "$(date -Iseconds) GATE $AGENT_ID PASS words=$WORDS facts=$FACTS" >> "$ORCH_LOG"
BATCH_PASSED=$((BATCH_PASSED + 1))
else
echo "$(date -Iseconds) GATE $AGENT_ID FAIL words=$WORDS facts=$FACTS" >> "$ORCH_LOG"
fi
done < <(comm -13 .deep_crawl/_pre_batch_files.txt .deep_crawl/_post_batch_files.txt)
echo "$(date -Iseconds) BATCH_DONE batch${BATCH_NUM} passed=${BATCH_PASSED}/${BATCH_TOTAL}" >> "$ORCH_LOG"
echo "" >> "$CRAWL_LOG"
# If any file fails: re-spawn the failing sub-agent with corrective instructions.
# Do NOT proceed to next batch until all findings files pass.
Thresholds are defined in .claude/skills/deep-crawl/configs/quality_gates.json under investigation_findings. If a file fails, log the re-spawn event and re-spawn the sub-agent:
echo " RE-SPAWN: $(basename $f .md) — reason: findings gate failed (${WORDS}w < 200w min, ${FACTS} < 5 FACT min)" >> "$CRAWL_LOG"
echo "$(date -Iseconds) RESPAWN $(basename $f .md) reason=\"density ${WORDS}w/${FACTS}FACT below floor\"" >> "$ORCH_LOG"
Re-spawn with: "Your output had {WORDS}w and {FACTS} [FACT] citations. Minimum is 200w and 5 [FACT]. Investigate deeper — read more source files, trace more hops, grep for more patterns."
Step 4: Checkpoint (mandatory). After each batch completes and passes the findings quality gate, update CRAWL_PLAN.md to mark completed tasks with [x]. Sub-agents never write to CRAWL_PLAN.md (concurrent writes would corrupt it).
# Mark completed tasks for batch N — the orchestrator does this, never sub-agents
# Replace P1/P2/P3/P4 with the actual priority prefix for the completed batch
for sentinel in .deep_crawl/batch_status/*.done; do
TASK_ID=$(basename "$sentinel" .done)
sed -i "s/^- \[ \] ${TASK_ID}/- [x] ${TASK_ID}/" .deep_crawl/CRAWL_PLAN.md
done
COMPLETED=$(grep -c '^\- \[x\]' .deep_crawl/CRAWL_PLAN.md)
echo "Checkpoint: $COMPLETED tasks completed"
echo "## Checkpoint (Batch $BATCH_NUM): $COMPLETED total completed" >> "$CRAWL_LOG"
Step 5: Handle failures. After a batch completes:
- Check which sentinel files are missing — those tasks failed
- Retry each failed task once (spawn a new sub-agent)
- If retry fails, log the task ID and continue — Phase 4's coverage check will catch gaps
- If ALL tasks in Batch 1 fail (Agent tool unavailable), switch to sequential fallback (see below)
Step 6: Coverage check. After all batches complete, verify the stopping criteria and coverage checks below. If coverage is insufficient, spawn Batch 5 with targeted gap-filling tasks.
Sequential Fallback
If the Agent tool is unavailable or all Batch 1 spawns fail, execute the crawl plan sequentially using the protocols below directly. This produces shallower results but ensures the pipeline completes. Log a warning: "Running in sequential fallback mode — investigation depth will be reduced."
Investigation Protocols
Sub-agents receive these protocol instructions verbatim:
Protocol A: Request Trace
1. Read the entry point function (full source)
2. Identify the first call to another module
3. Read that function (full source)
4. Repeat until terminal side effect (no hop limit — follow the full chain)
5. Record in .deep_crawl/findings/traces/{NN}_{name}.md:
entry_function (file:line)
→ called_function (file:line) — what it does in 1 sentence
→ next_function (file:line) — what it does in 1 sentence
→ [SIDE EFFECT: db.commit] (file:line)
6. Note branching (error paths, conditional logic) at each hop
7. Note data transformations between hops
8. Note gotchas discovered during tracing
9. [ABSENCE] check: at each hop, note expected-but-missing elements — missing error handling, missing validation, missing logging, missing auth checks, missing null checks. Tag each as [ABSENCE] with grep confirmation.
**TypeScript-specific trace guidance:**
When tracing through TypeScript/JavaScript codebases:
- **Async boundaries:** Note where sync→async transitions occur (missing `await`, `.then()` chains). Flag any function that starts a Promise chain without error handling.
- **Middleware chains:** In Express/Fastify/Koa, trace through the middleware stack in order. Note where `next()` is called, where it's skipped (early return), and where error middleware breaks the chain.
- **DI resolution:** In NestJS/Angular, trace from `@Injectable()` declaration through `@Inject()` sites. Note lifecycle hooks (`onModuleInit`, `onApplicationBootstrap`) that run at startup.
- **Barrel file hops:** When a trace passes through an `index.ts` that only re-exports, note this but don't count it as a meaningful hop. The real target is the source module.
- **Type narrowing at branches:** Note where `if (x instanceof Foo)` or discriminated union checks (`if (x.type === 'bar')`) gate code paths — these are the real branching points.
- **Event-driven flows:** When the trace reaches `emit()`, `on()`, or `subscribe()`, follow the event to ALL listeners. Note if multiple handlers exist for the same event.
Protocol B: Module Deep Read
1. Read the entire module
2. Write to .deep_crawl/findings/modules/{module_name}.md:
- What this module does (1-2 sentences of BEHAVIOR)
- What's non-obvious about it
- What breaks if you change it (blast radius)
- What it depends on at runtime (not just imports)
- For each public function/method:
- 1-sentence behavioral description
- Preconditions
- Side effects
- Error behavior
2b. [ABSENCE] check: for each public function, note expected-but-missing elements — missing docstring on complex method, missing type hints on public API, missing validation on inputs from external boundaries, missing error handling on I/O operations. Tag each as [ABSENCE].
2c. Check for corresponding test files:
- Python: Look for tests/test_{module_name}.py, tests/unit/**/test_{module_name}.py,
tests/{module_name}_test.py, and any test file importing from this module
- TypeScript/JavaScript: Look for {module_name}.test.ts, {module_name}.spec.ts,
__tests__/{module_name}.ts, and in-source tests via `import.meta.vitest` blocks
within the module itself (colocated testing pattern)
- If found, scan test function names and docstrings/descriptions
- Note which public functions have test coverage and which don't
- Add to findings:
Test coverage: {tested_count}/{total_public} public functions tested.
Tested: {list}. Untested: {list}.
Test file: {path} ({N} test functions)
3. Record any gotchas
**TypeScript-specific module investigation:**
When deep-reading a TypeScript module, also document:
- **Exported types:** List all `export interface`, `export type`, `export enum` — these are the module's contract. Note which are used externally (via xray import graph).
- **Decorator metadata:** For classes with decorators (`@Entity`, `@Injectable`, `@Controller`), extract the decorator arguments — they encode runtime configuration that isn't visible from the type signature alone.
- **Generic constraints:** Note generic type parameters with constraints (`T extends SomeBase`) — these restrict what callers can pass and are a common source of type errors when changed.
- **`any` escape hatches:** Count `as any`, `@ts-ignore`, `@ts-expect-error` in the module. Each is a place where the type system has been deliberately bypassed — potential bug hiding spots.
- **Re-export surface:** If the module is a barrel file (mostly re-exports), note what it re-exports and whether it adds any transformation.
Protocol C: Cross-Cutting Concern
1. Grep for relevant patterns across the entire codebase
2. Categorize the results
3. Read representative examples in full — at least max(3, result_count / 5) examples, sampling from different subsystems
4. Identify the dominant strategy
5. Flag deviations from the dominant strategy
6. [ABSENCE] check: for the concern being investigated, grep for expected-but-missing implementations. E.g., for error handling: modules with no try/except around I/O; for config: env vars read without defaults; for auth: routes without auth middleware. Tag each as [ABSENCE] with grep evidence.
7. Write to .deep_crawl/findings/cross_cutting/{concern_name}.md
Grep patterns for common concerns (language-adaptive):
Use $CODE_LANG from Phase 0b pre-flight to select patterns. All grep commands should target $DEEP_CRAWL_ROOT.
Python patterns (use when CODE_LANG=python):
grep -rn "except " --include="*.py" "$DEEP_CRAWL_ROOT" | head -40
grep -rn "except.*pass" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "retry\|backoff\|fallback" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "os.getenv\|os.environ" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "config\[" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "^[a-z_].*= " --include="*.py" "$DEEP_CRAWL_ROOT" | grep -v "def \|class \|#\|import " | head -30
grep -rn "_instance\|_cache\|_registry\|_pool" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "global " --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "asyncio\.run\|loop\.run_until_complete" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "async def " --include="*.py" "$DEEP_CRAWL_ROOT" | wc -l
grep -rn "pickle\.loads\|pickle\.load\|yaml\.load\|marshal\.loads" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "class.*Exception\|class.*Error" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "except.*pass\|except:$" --include="*.py" "$DEEP_CRAWL_ROOT"
grep -rn "raise " --include="*.py" "$DEEP_CRAWL_ROOT" | head -40
TypeScript/JavaScript patterns (use when CODE_LANG=typescript):
grep -rn "catch" --include="*.ts" --include="*.tsx" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -40
grep -rn "catch\s*{}" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules # empty catch — high value
grep -rn "retry\|backoff\|fallback" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules
grep -rn "process\.env" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -30
grep -rn "config\.\|getConfig\|loadConfig" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -20
grep -rn "^let \|^var " --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -30
grep -rn "cache\|singleton\|_instance\|getInstance" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -20
grep -rn "async function\|async (" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | wc -l
grep -rn "new Promise\|Promise\.all" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -20
grep -rn "eval(\|new Function(" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules
grep -rn "JSON\.parse" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -20
grep -rn "as any\|@ts-ignore\|@ts-expect-error" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -20
grep -rn "extends Error" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules
grep -rn "throw " --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -40
grep -rn "process\.exit" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules
grep -rn "readFile\|readdir\|createReadStream" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -20
TypeScript deep patterns (run after the basic patterns above when time allows):
# Promise error handling gaps
grep -rn "\.then(" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | grep -v "\.catch" | head -20 # .then() without .catch()
grep -rn "Promise\.all\b" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules # bulk operations — check error handling
grep -rn "new Promise" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -15 # manual Promise construction — often buggy
# Barrel file analysis (circular import risk)
for f in $(find "$DEEP_CRAWL_ROOT" -name "index.ts" -not -path "*/node_modules/*" 2>/dev/null); do
EXPORTS=$(grep -c "^export" "$f" 2>/dev/null || echo 0)
LOGIC=$(grep -cvE "^export|^import|^$|^//" "$f" 2>/dev/null || echo 0)
if [ "$EXPORTS" -gt 3 ] && [ "$LOGIC" -lt 5 ]; then echo "BARREL: $f ($EXPORTS re-exports)"; fi
done
# Type safety escape hatches (aggregate)
echo "=== Type safety signals ==="
echo "as any: $(grep -rn 'as any' --include='*.ts' "$DEEP_CRAWL_ROOT" | grep -v node_modules | wc -l)"
echo "@ts-ignore: $(grep -rn '@ts-ignore' --include='*.ts' "$DEEP_CRAWL_ROOT" | grep -v node_modules | wc -l)"
echo "@ts-expect-error: $(grep -rn '@ts-expect-error' --include='*.ts' "$DEEP_CRAWL_ROOT" | grep -v node_modules | wc -l)"
echo "non-null assertion (!.): $(grep -rn '\!\.' --include='*.ts' "$DEEP_CRAWL_ROOT" | grep -v node_modules | wc -l)"
# AbortController / cancellation patterns
grep -rn "AbortController\|AbortSignal\|signal:" --include="*.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules | head -10
# Module augmentation (global type patching — coupling risk)
grep -rn "declare module\|declare global" --include="*.ts" --include="*.d.ts" "$DEEP_CRAWL_ROOT" | grep -v node_modules
# Workspace/monorepo structure
ls "$DEEP_CRAWL_ROOT"/packages/*/package.json 2>/dev/null && echo "MONOREPO: packages/* detected"
ls "$DEEP_CRAWL_ROOT"/apps/*/package.json 2>/dev/null && echo "MONOREPO: apps/* detected"
cat "$DEEP_CRAWL_ROOT/package.json" 2>/dev/null | grep -A5 '"workspaces"' | head -8
Exception taxonomy investigations (Protocol C):
When the crawl plan includes an exception taxonomy task, Protocol C agents should:
- Find all classes inheriting from Exception/BaseException (Python) or Error (TypeScript)
- Build inheritance tree (which exceptions/errors derive from which)
- For each custom exception/error: where it's raised/thrown, where it's caught, whether
it crosses module boundaries
- Identify uncaught exception paths (raise/throw without corresponding catch in callers)
- Identify silent failure patterns:
except: pass (Python), empty catch {} (TypeScript),
log-and-swallow — these are high-value gotcha candidates
- Write to findings/cross_cutting/exception_taxonomy.md
Protocol D: Convention Documentation
1. Read examples of the pattern from xray pillar/hotspot list — at least max(5, pillar_count / 3) examples, covering different architectural layers
2. Identify the common structure
3. State the convention as a directive ("always X", "never Y")
4. Read each flagged deviation
5. Assess: intentional variation or oversight?
6. [ABSENCE] check: for each convention identified, grep for expected adherence in untested modules. Note modules that should follow the convention but don't. Tag as [ABSENCE].
7. Write to .deep_crawl/findings/conventions/patterns.md
**TypeScript-specific conventions to document:**
When documenting conventions for TypeScript codebases, check for these patterns:
- **Import organization:** Are imports grouped (external → internal → types)? Are path aliases used consistently? Are barrel files the standard import target?
- **Null safety style:** Optional chaining (`?.`) vs explicit null checks? Non-null assertions (`!`) — banned or accepted?
- **Error class hierarchy:** One base `AppError` or ad-hoc `throw new Error()`? Error codes or just messages?
- **Async patterns:** Always `async/await` or mixed with `.then()`? Raw Promises or wrapped (e.g., `Result<T, E>` monads)?
- **Type annotation style:** Return types always explicit or inferred? `interface` vs `type` preference? `unknown` vs `any` for untyped data?
- **Component patterns (if React/Vue):** Function components vs class? Props interface naming (`FooProps`)? Hook extraction conventions?
- **Decorator usage (if NestJS/Angular):** Custom decorator conventions? Metadata patterns?
Protocol E: Reverse Dependency & Change Impact
1. Read xray reverse dependency data for the assigned hub module cluster:
- .imports.graph[module]["imported_by"] — list of importer modules
- .imports.distances.hub_modules — sorted by connection count
- .calls.reverse_lookup[function] — {callers, caller_count, module_count, impact_rating}
- .calls.high_impact — pre-filtered high-impact functions
2. For each hub module in the cluster, read 2-3 representative callers (full source)
3. Identify:
- Which callers depend on specific function signatures
- Which callers depend on specific return value shapes
- Which callers depend on specific side effects or state mutations
4. Classify changes as safe vs dangerous:
- Safe: internal refactoring, adding optional parameters, performance changes
- Dangerous: signature changes, return type changes, side effect changes, exception changes
5. [ABSENCE] check: for each hub module, note missing safeguards — callers that don't handle the hub's error cases, importers with no tests covering their usage of the hub, type contracts that aren't enforced at the boundary. Tag as [ABSENCE].
6. Write to .deep_crawl/findings/impact/{cluster_name}.md:
For each hub module:
- Module name and importer count
- High-impact functions with caller counts and impact ratings
- Signature-change consequences (which callers break and how)
- Behavior-change consequences (which callers produce wrong results)
- Safe changes list
- Dangerous changes list with specific blast radius
Protocol F: Change Scenario Walkthrough
1. Read Protocol E impact findings for context on hub modules and blast radii
2. Read Protocol B module findings for behavioral detail
3. Read conventions findings for coding patterns to follow
4. Derive scenarios from domain profile's primary_entity + Extension Points findings:
- "Add new {primary_entity}" (always include this scenario)
- "Modify existing {primary_entity} behavior" (always include — traces the change path through existing wiring, identifies what to update vs what to leave alone)
- "Change a data model field" (always include if >=3 domain entities detected — shows the full propagation: model → serialization → consumers → tests → migrations)
- "Modify {top hub module} behavior" (top 3 hub modules by connection count)
- "Add new external dependency" (if >3 external deps detected)
5. For each scenario, build a step-by-step checklist:
- Ordered steps with file:line targets
- What to create/modify at each step
- Validation commands (test commands, grep checks)
- Common mistakes (derived from gotchas and impact analysis)
- [ABSENCE] check: for each step, note missing safeguards — steps where there's no test to verify correctness, no migration script, no rollback path. Tag as [ABSENCE].
6. Write to .deep_crawl/findings/playbooks/{scenario_name}.md
Checkpoint discipline: After each batch completes, update CRAWL_PLAN.md to mark completed tasks. In sequential fallback mode, update after every 5 completed tasks.
When to stop crawling — ALL must be true:
- All Priority 1 (request traces) tasks are complete
- All Priority 2 (high-uncertainty modules) are read
- All Priority 3 (pillar behavioral summaries) are complete
- All Priority 4 (cross-cutting concerns) are investigated
- All Priority 5 (conventions and patterns) are documented
- All Priority 6 (gap investigation) tasks are attempted
- All Priority 7 (change impact analysis) tasks are complete
- Current findings can answer all 12 standard questions (see below)
- Coverage check passes (see below)
Coverage check — ALL must be true:
(Note: This check verifies the Phase 1 coverage pre-check was effective. If gaps remain, the pre-check heuristic needs improvement.)
- At least 50% of subsystems/top-level packages have at least one module deep-read
- Number of modules deep-read >= max(10, file_count / 40)
- Number of request traces >= number of entry points identified by xray
- Every cross-cutting concern has examples from at least 3 different subsystems
- At least one impact card exists for each hub module cluster
- At least one change playbook exists
Phase 3: ASSEMBLE (Orchestrated Section Production)
echo "## Phase 3: ASSEMBLE — $(date -Iseconds)" >> "$CRAWL_LOG"
echo "$(date -Iseconds) PHASE 3 ASSEMBLE" >> "$ORCH_LOG"
Phase 3 delegates assembly to parallel sub-agents, each producing specific template sections from their assigned subset of findings. The orchestrator's role is: spawn agents, monitor sentinels, concatenate section files. Do NOT read findings content yourself.
Value Hierarchy (shared with all assembly sub-agents)
- Tier 1 (MUST include, never cut): Bugs, security issues, data loss risks, crashes, dead code, broken features
- Tier 2 (MUST include, cut only if literally duplicated): Behavioral gotchas, non-obvious side effects, error handling deviations, initialization order dependencies, state leakage risks
- Tier 3 (include unless redundant with another finding): Patterns with N/M counts, conventions, structural facts, configuration details
- Tier 4 (include unless literally duplicated elsewhere in this document; reference xray for raw structural dumps only): Class skeletons, import graphs, file listings
Phase 4 may merge Tier 3-4 entries that describe the same fact, but must not drop distinct findings. When in doubt, keep both. Phase 4 may NEVER cut Tier 1-2 content.
Step 0: Concatenate findings (word count reference for retention check)
cat .deep_crawl/findings/calibration/*.md \
.deep_crawl/findings/traces/*.md \
.deep_crawl/findings/modules/*.md \
.deep_crawl/findings/cross_cutting/*.md \
.deep_crawl/findings/conventions/*.md \
.deep_crawl/findings/impact/*.md \
.deep_crawl/findings/playbooks/*.md \
> .deep_crawl/SYNTHESIS_INPUT.md 2>/dev/null
wc -w .deep_crawl/SYNTHESIS_INPUT.md
Record the total word count — you will use it to verify retention in Step 8.
Step 1: Pre-extract gotchas via bash
No sub-agent needed. Extract formal gotcha sections from all findings so S5 has them consolidated:
for f in .deep_crawl/findings/{traces,modules,cross_cutting,conventions,impact,playbooks}/*.md; do
gotcha_line=$(grep -n "^## Gotcha\|^### Gotcha" "$f" | head -1 | cut -d: -f1)
if [ -n "$gotcha_line" ]; then
echo "### From: $(basename "$f" .md)"
tail -n +"$gotcha_line" "$f"
echo ""
fi
done > .deep_crawl/sections/gotcha_extracts.md
Step 1a: Extract state diagrams from trace findings
No sub-agent needed. Mechanically grep trace findings for state transition patterns and generate mermaid state diagrams if found:
# Look for state transition patterns in trace findings
STATE_PATTERNS=$(grep -rn "state\|status\|phase\|stage\|transition\|→.*→" \
.deep_crawl/findings/traces/*.md 2>/dev/null | head -20)
if [ -n "$STATE_PATTERNS" ]; then
echo "## State Diagrams" > .deep_crawl/sections/state_diagrams.md
echo "" >> .deep_crawl/sections/state_diagrams.md
echo "State transitions extracted from request traces:" >> .deep_crawl/sections/state_diagrams.md
echo "" >> .deep_crawl/sections/state_diagrams.md
echo '```mermaid' >> .deep_crawl/sections/state_diagrams.md
echo "stateDiagram-v2" >> .deep_crawl/sections/state_diagrams.md
# Assembly agent S1 will populate the actual transitions
echo " [*] --> TODO_POPULATE_FROM_TRACES" >> .deep_crawl/sections/state_diagrams.md
echo '```' >> .deep_crawl/sections/state_diagrams.md
echo "State patterns found — S1 assembly agent will include in Critical Paths section header."
else
echo "No state transition patterns found in traces."
touch .deep_crawl/sections/state_diagrams.md # empty file
fi
If state_diagrams.md has content, assembly agent S1 includes it at the top of the Critical Paths section.
Step 1b: Temporarily remove CLAUDE.md compression contamination
Sub-agents spawn with fresh context and read CLAUDE.md files automatically. If the project's CLAUDE.md references "compressed intelligence" or similar framing, sub-agents absorb "my job is compression" before reading their task prompt. Temporarily rename CLAUDE.md files to prevent this:
# Rename project CLAUDE.md (and parent directory's if it exists)
ROOT_PATH="${DEEP_CRAWL_ROOT:-$(pwd)}"
[ -f "$ROOT_PATH/CLAUDE.md" ] && mv "$ROOT_PATH/CLAUDE.md" "$ROOT_PATH/CLAUDE.md.assembly_save"
PARENT_PATH=$(dirname "$ROOT_PATH")
[ -f "$PARENT_PATH/CLAUDE.md" ] && mv "$PARENT_PATH/CLAUDE.md" "$PARENT_PATH/CLAUDE.md.assembly_save"
IMPORTANT: Restore these files after all sub-agents complete (after Step 9). See Step 9 completion.
Step 1c: Generate structural skeleton
Before spawning assembly agents, mechanically generate a skeleton of prescribed ### headers from investigation findings. This skeleton locks section structure — assembly agents fill content under prescribed headers but cannot merge or remove them.
# === STRUCTURAL SKELETON GENERATION ===
SKEL=".deep_crawl/sections/_skeleton.md"
echo "# Assembly Skeleton — Prescribed Section Structure" > $SKEL
echo "# Assembly agents MUST produce every ### listed below for their section." >> $SKEL
echo "" >> $SKEL
# S1: Critical Paths — one ### per trace finding
echo "## S1: Critical Paths" >> $SKEL
N=1
for f in .deep_crawl/findings/traces/*.md; do
[ -f "$f" ] || continue
[[ "$(basename "$f")" == "00_calibration.md" ]] && continue
TITLE=$(head -1 "$f" | sed 's/^#* *//' | cut -c1-80)
echo "### Path $N: $TITLE" >> $SKEL
N=$((N+1))
done
echo "" >> $SKEL
# S2: Module Behavioral Index — one ### per module finding
echo "## S2: Module Behavioral Index" >> $SKEL
for f in .deep_crawl/findings/modules/*.md; do
[ -f "$f" ] || continue
[[ "$(basename "$f")" == "00_calibration.md" ]] && continue
# Extract module path from first line or filename
MOD=$(head -3 "$f" | grep -oP '`[^`]+\.py`' | head -1)
[ -z "$MOD" ] && MOD="\`$(basename "$f" .md).py\`"
echo "### $MOD -- Detailed Behavioral Analysis" >> $SKEL
done
echo "" >> $SKEL
# S3b: Error Handling — one ### per strategy/pattern in cross-cutting findings
echo "## S3b: Error Handling" >> $SKEL
if [ -f .deep_crawl/findings/cross_cutting/error_handling.md ]; then
grep '^## \|^### ' .deep_crawl/findings/cross_cutting/error_handling.md \
| sed 's/^## /### /' | sed 's/^### ### /### /' >> $SKEL
fi
echo "" >> $SKEL
# S5: Gotchas — domain cluster headers from module directory groupings
echo "## S5: Gotchas" >> $SKEL
ls .deep_crawl/findings/modules/*.md 2>/dev/null \
| xargs -I{} head -3 {} \
| grep -oP '`[^`/]+/' \
| sort -u | tr -d '`' \
| while read -r pkg; do
# Convert package dir to readable cluster name
CLUSTER=$(echo "$pkg" | sed 's|/$||' | sed 's|_| |g' | sed 's|\b\(.\)|\u\1|g')
echo "### ${CLUSTER} Gotchas" >> $SKEL
done
# Always include a cross-cutting cluster
echo "### Cross-Cutting Gotchas" >> $SKEL
echo "" >> $SKEL
echo "Skeleton generated: $(grep -c '^### ' $SKEL) subsections prescribed"
The skeleton is informational, not rigid for every section. S1 and S2 MUST follow it exactly (one ### per finding file). S3b and S5 use it as guidance — they may adjust ### headers based on content but should maintain similar granularity.
Step 1d: Verify skeleton exists (blocking gate)
test -f .deep_crawl/sections/_skeleton.md || { echo "HALT: Skeleton not generated. Run Step 1c first."; exit 1; }
SKEL_H3=$(grep -c "^### " .deep_crawl/sections/_skeleton.md)
echo "Skeleton has $SKEL_H3 prescribed subsections (min: 25)"
[ "$SKEL_H3" -lt 25 ] && echo "WARN: Skeleton has fewer than 25 subsections — investigation may be shallow"
Do NOT proceed to Step 2 if the skeleton is missing. This prevents assembly agents from producing unstructured output.
Step 2: Spawn S1, S2, S3a, S3b, S4 in parallel
Before spawning each assembly agent, apply the agent logging protocol: write the prompt to .deep_crawl/agent_logs/prompts/S{N}.md and log the SPAWN event. After each assembly agent completes, write the return text to .deep_crawl/agent_logs/results/S{N}.md and log the DONE event with output file stats.
Launch 5 assembly sub-agents simultaneously (all with run_in_background: true):
| Agent | Sections | Input Files | Est. Input |
|---|
| S1 | Critical Paths | findings/traces/*.md | ~11K words |
| S2 | Module Behavioral Index, Domain Glossary | findings/modules/*.md | ~25K words (use chunked writes — see protocol below) |
| S3a | Key Interfaces | findings/modules/*.md (public API extraction) + findings/cross_cutting/agent_communication.md | ~12K words |
| S3b | Error Handling, Shared State | findings/cross_cutting/{error_handling,initialization,shared_state,database_storage,async_boundaries,exception_taxonomy}.md + findings/modules/*.md (grep for error/exception/retry patterns) | ~16K words |
| S4 | Configuration Surface, Conventions | findings/cross_cutting/{configuration,env_dependencies,llm_providers}.md + findings/modules/config.md + findings/conventions/*.md | ~16K words |
Note: The file lists above are illustrative examples. Adjust filenames to match actual findings on disk. Use ls .deep_crawl/findings/{category}/ to discover actual filenames before constructing prompts.
Each sub-agent prompt must be self-contained:
You are assembling investigation findings into onboarding document sections. Include every finding — do not summarize, condense, or compress.
## Your Task
Produce these template sections: [SECTION NAMES]
## Input
Read ONLY these files: [FILE PATHS]
## Template Format
[Copy the relevant section templates verbatim from .claude/skills/deep-crawl/templates/DEEP_ONBOARD.md.template]
## Structural Contract
Read .deep_crawl/sections/_skeleton.md for your section's prescribed ### headers.
- For Module Behavioral Index and Critical Paths: each prescribed ### MUST appear in your output exactly as listed. Each findings file maps to one ###. Do not merge.
- For Error Handling and Gotchas: use the skeleton's ### headers as guidance. Maintain similar granularity but adjust header wording based on actual content.
- For other sections: no skeleton constraint — derive structure from content.
- You MAY add additional ### subsections beyond the skeleton's prescriptions.
## Section Hierarchy Rule
Your output MUST use exactly one `## ` header per template section assigned to you (e.g., `## Error Handling Strategy`, `## Module Behavioral Index`). All subdivisions within a section MUST use `### ` or deeper. Never promote subsection content to `## ` level — a flat document with many `## ` headers destroys navigability. If your template section has subsections, they are `### `. If those subsections have further divisions, they are `#### `.
## S2 Historical Risk Annotation
If you are S2 (Module Behavioral Index): for each module ### being assembled, check if it appears in `git.risk`, `git.function_churn`, or `git.velocity` from `output/$REPO_NAME/data/xray.json`. If it does, append a brief **Historical Risk** note at the end of that module's subsection with: risk score, most-volatile functions, and trend direction. Format: `> **Git risk:** 0.88 — volatile functions: load_config (8 commits, 2 hotfixes). Trend: stable.`
## S3b Depth Directive
If you are S3b (Error Handling, Shared State): the Error Handling Strategy section must document the dominant error pattern, per-subsystem deviations, retry strategies, exception hierarchies, and recovery paths. Target: >= 3,500 words for Error Handling alone. Read module findings for error/exception/retry patterns in addition to cross-cutting findings — every module's error handling deviations contribute to this section. If `findings/cross_cutting/exception_taxonomy.md` exists, include it as a subsection under Error Handling: inheritance tree of custom exceptions, raise/catch mapping, uncaught paths, and silent failure patterns (`except: pass`, bare except, log-and-swallow). Silent failures are high-value gotcha candidates — flag them in your gotchas output file.
## Rules
- INCLUDE EVERY FINDING. The output context window is 1M tokens. Your section will consume less than 5% of available context. There is no reason to drop, summarize, or condense any finding.
- The ONLY reason to exclude a finding is if it is literally stated elsewhere in your section or is derivable from the module's file name alone.
- Place each finding at full fidelity with its original [FACT], [PATTERN], or [ABSENCE] tag and file:line citation.
- When multiple findings describe the same code from different angles, include ALL of them — each angle provides context the others miss.
- Write conventions as directives — "Always X", "Never Y".
- Your output should be 80-100% of your input word count. If you produce less than 80%, you have dropped findings. Re-read your input and find what you missed.
## Quality Floor (mechanically checked after you finish)
- Citation density floor varies by section type — see quality_gates.json density_tiers.
High-evidence sections (impact, gotchas, contracts): 3.0/100w.
Medium (module index, interfaces, playbooks): 2.0/100w.
Narrative (critical paths, conventions): 1.0/100w.
Structural (glossary, reading order): no floor.
- Word count: >= 80% of your input findings word count
If your section fails, you will be re-spawned.
For format guidance, see .claude/skills/deep-crawl/configs/exemplar_templates.md.
For quality reference from this repo, see .deep_crawl/findings/calibration/cal_{type}.md.
## Secondary Output
Collect every gotcha/warning/danger note encountered → write to:
.deep_crawl/sections/gotchas_from_S{N}.md
## Output
Write each section to its own file: .deep_crawl/sections/{section_name}.md
(e.g., critical_paths.md, module_index.md, domain_glossary.md)
Sentinel: touch .deep_crawl/sections/S{N}.done
## Large Output — Chunked Write Protocol
If your assembled section exceeds ~25,000 words, you MUST write it in chunks to avoid
hitting output token limits:
1. Write the first ~20,000 words to `.deep_crawl/sections/{section_name}.md` (creates the file)
2. Write each subsequent ~20,000 word chunk by reading back the current file content's
last few lines, then appending via a Bash command:
`cat >> .deep_crawl/sections/{section_name}.md << 'CHUNK_EOF'`
followed by the chunk content and `CHUNK_EOF`
3. After all chunks are written, touch the sentinel file
This is critical for S2 (Module Behavioral Index) which often exceeds 25K words on large repos.
Do NOT attempt to write the entire section in a single Write call if it exceeds 25K words.
## Execution Log (assembly agents only)
Before your FINAL CHECK, write a brief execution log to `.deep_crawl/agent_logs/results/S{N}.md`:
- Input: {count} files read, {total_words} words of input
- Output: {list of files written with word counts}
- Sections produced: {list}
- Key structural decisions (e.g., "merged 3 small modules into one ### subsection")
- Issues encountered (e.g., "finding file X was empty", "skeleton prescribed ### for module not in findings")
## FINAL CHECK (do this before writing sentinel)
Before touching your sentinel file, verify your output:
1. Run: `grep -c "^## " <your_output_file>` — MUST be exactly 1 per section you wrote
2. Run: `grep -c "^# " <your_output_file>` — MUST be exactly 0
If either check fails, fix your headers before writing the sentinel.
A section with multiple `## ` headers or any `# ` headers will be rejected by the quality gate.
## Constraints
- Read-only: never modify source code
- Do NOT spawn sub-agents yourself
Step 3: Monitor S1–S4 completion
Check for sentinel files. All 5 must complete before proceeding:
ls .deep_crawl/sections/S{1,2,3a,3b,4}.done 2>/dev/null | wc -l # expect 5
S2 output-limit recovery: If S2 fails with "exceeded output token maximum" or produces
no sentinel after other agents complete, the module index is too large for a single agent.
Split into S2a + S2b:
- S2a: First half of
findings/modules/*.md files (alphabetically) → module_index.md
- S2b: Second half → append to
module_index.md via cat >> .deep_crawl/sections/module_index.md
- S2b also writes:
domain_glossary.md (small, derived from all module findings)
- Both use the same prompt template but with split file lists. S2b's prompt includes:
"Append your output to the existing module_index.md file using: cat >> .deep_crawl/sections/module_index.md"
Re-spawn with run_in_background: true. Wait for both S2a.done and S2b.done.
Step 4: Spawn S5 AND S6 in parallel (depend on S1-S3a-S3b-S4 gotcha outputs)
Before spawning S5 and S6, apply the agent logging protocol: write prompts to .deep_crawl/agent_logs/prompts/S5.md and .deep_crawl/agent_logs/prompts/S6.md, log SPAWN events. After completion, write return text to results and log DONE events.
Launch S5 and S6 simultaneously (both with run_in_background: true):
| Agent | Sections | Input Files | Est. Input |
|---|
| S5 | Gotchas, Hazards, Extension Points, Reading Order | sections/gotcha_extracts.md + sections/gotchas_from_S{1,2,3a,3b,4}.md + CRAWL_PLAN.md (for structure) | ~14K words |
| S6 | Change Impact Index, Data Contracts, Change Playbooks | findings/impact/*.md + findings/playbooks/*.md + findings/modules/*.md (grep for Pydantic/dataclass) + sections/state_diagrams.md | ~15K words (use chunked writes if output exceeds 25K words) |
S5's prompt follows the same template as S1-S4, with these additions for the Gotchas section:
Organize gotchas into domain-cluster ### subsections derived from the investigation's subsystem structure (e.g., "Agent System", "Data Models", "Execution Engine", "LLM Integration", "Configuration"). Derive cluster names from the module findings directory — group findings/modules/*.md by top-level package directory. Within each cluster, order entries by severity (critical first). Prefix each entry with severity tag: [CRITICAL], [HIGH], or [MEDIUM]. Target: one ### per 5-15 gotchas. Never put more than 15 gotchas under a single ### heading.
Severity calibration (mandatory): Read .claude/skills/deep-crawl/configs/quality_gates.json field gotcha_severity_criteria before assigning severity tags. Apply these rules strictly:
- CRITICAL = active data loss, security vulnerability, or silent corruption in production NOW. Not "could cause problems" — "is causing problems." Expect <=5% of all gotchas to be CRITICAL.
- HIGH = behavioral surprise that causes bugs during common modification tasks. Dormant until code nearby is changed.
- MEDIUM = confusing or misleading code that wastes time but doesn't break production.
After tagging all gotchas, run a self-check: count CRITICAL gotchas. If CRITICAL count > total_gotchas * 0.05, review each CRITICAL and demote any that are "potential problem" rather than "active problem" to HIGH. Log: Severity calibration: {N} CRITICAL, {M} HIGH, {K} MEDIUM ({N/total}% CRITICAL).
Root-cause clustering (mandatory): After severity tagging, identify gotchas that share a common root cause — same design decision, same pattern, same underlying assumption. For each cluster of >=2 related gotchas:
- Add
*Root cause: {1-sentence description}* after the cluster's ### heading
- Cross-link related gotchas:
(related: #{N} — same root cause)
- If related gotchas span different domain clusters, add the cross-link across clusters
This surfaces patterns like "these 5 gotchas all stem from the dual Pydantic/SQLAlchemy model pattern" rather than treating each as independent.
S5's template sections remain: Gotchas, Hazards — Do Not Read, Extension Points, and Reading Order from DEEP_ONBOARD.md.template.
TypeScript-specific gotcha categories (ensure these are checked during gotcha clustering):
When organizing gotchas for TypeScript projects, ensure these categories are represented if evidence exists:
- Type safety escapes:
as any casts, @ts-ignore directives, non-null assertions (!) hiding potential null access
- Async error handling: Missing
.catch() on Promise chains, unhandled rejection paths, try/catch not wrapping await
- Circular imports: Barrel files (
index.ts) creating import cycles, runtime undefined from circular dependencies
- Build/runtime divergence: Type-only imports stripped at build time,
const enum inlining differences, path alias resolution mismatches between IDE and runtime
- State management: Module-level
let used as cache without invalidation, singleton services with mutable state, React useState without cleanup
- External boundary unsafety:
JSON.parse() returning any, API response types asserted but not validated at runtime, process.env values used without undefined checks
S6's prompt for Data Contracts:
- For Python: extract all Pydantic models, dataclasses, TypedDicts, and NamedTuples from module findings + xray
investigation_targets.domain_entities.
- For TypeScript: extract all interfaces, type aliases (especially discriminated unions), Zod schemas, class-validator decorated classes, and enums. Also check xray
ts_interfaces and ts_type_aliases data.
- Format as table with Model/Interface, File, Type (interface/class/enum/schema), Key Fields, Validation (Zod/class-validator/none), Gotcha.
Field-level deep trace (mandatory for top contracts): For the top 3-5 contracts by importer count (from xray import graph), produce a Cross-Boundary Flow Analysis subsection with field-level specificity:
- Show the exact fields at creation (producer side) with types and defaults
- Show each transformation boundary — where fields are added, dropped, renamed, or re-typed
- Show the final shape at consumption (consumer side)
- Flag field divergence: where the same logical entity has different field sets in different representations (e.g., Pydantic model has 12 fields, SQLAlchemy model has 10,
to_dict() emits 8)
- Include
[FACT] citations at each boundary: created at {file}:{line}, transformed at {file}:{line}, consumed at {file}:{line}
This depth is what allows a downstream agent to implement a new consumer without reading source. Summary-level tables ("Model X has fields a, b, c") are insufficient — the value is in the boundary transformations.
S6's prompt for Change Impact Index: organize impact findings by hub module cluster. Each cluster gets a table showing importers, high-impact functions, signature-change consequences, behavior-change consequences, safe vs dangerous changes. Also read git.coupling_clusters from output/$REPO_NAME/data/xray.json. For each cluster, add a Hidden Coupling row to the relevant hub module's impact table. Format: When modifying {file_a}, also verify {file_b} — {count} historical co-changes with no import relationship.
S6's prompt for Change Playbooks: organize playbook findings into step-by-step checklists with validation commands and common mistakes. Protocol F now produces both "add new" and "modify existing" playbooks — ensure both types are assembled. Modification playbooks are especially valuable because they trace through existing wiring (what to change, what to leave alone, what breaks silently) rather than creating from scratch.
S6's prompt must include this Playbook Quality Floor (mechanically checked after you finish):
## Playbook Quality Floor (each playbook individually)
- Minimum 800 words
- Minimum 30 [FACT] citations
- Minimum 8 common mistakes with behavioral explanations
- Citation density: >= 3.0 [FACT] per 100 words
If any playbook fails, you will be re-spawned.
For format guidance, see .claude/skills/deep-crawl/configs/exemplar_templates.md.
For quality reference from this repo, see .deep_crawl/findings/calibration/cal_{type}.md.
Thresholds are defined in .claude/skills/deep-crawl/configs/quality_gates.json under playbooks.
S6's prompt for Change Playbooks quality checks:
- Each playbook individually: >= 800 words, >= 30 [FACT] citations
- Each playbook must have: 8-10 common mistakes with behavioral explanations
- Citation density per playbook: >= 3.0 [FACT] per 100 words
- If a reference playbook exists (e.g., modify_llm_provider.md), use it as the quality bar
Step 5: Monitor S5+S6 completion
ls .deep_crawl/sections/S{5,6}.done 2>/dev/null | wc -l # expect 2
Step 5b: SECTION QUALITY GATE (mandatory before concatenation)
After all assembly sub-agents (S1-S6) complete, run this mechanical check. The orchestrator MUST NOT write section content itself — if a section is missing or below quality, re-spawn the appropriate assembly sub-agent. The only sections the orchestrator writes directly are the small metadata sections listed in Step 6.
# === SECTION QUALITY GATE (mandatory before concatenation) ===
# (a) Existence check
MISSING=""
for section in critical_paths module_index change_impact_index \
key_interfaces data_contracts error_handling shared_state \
config_surface conventions gotchas hazards extension_points; do
[ -f ".deep_crawl/sections/${section}.md" ] || MISSING="$MISSING $section"
done
[ -n "$MISSING" ] && echo "GATE FAIL — missing sections:$MISSING"
# (b) Citation density check — tiered by section type
get_density_floor() {
case "$1" in
change_impact_index|gotchas|data_contracts) echo 3 ;;
module_index|key_interfaces|shared_state|change_playbooks|error_handling|hazards) echo 2 ;;
critical_paths|config_surface|conventions|extension_points) echo 1 ;;
*) echo 0 ;; # domain_glossary, reading_order, environment_bootstrap, etc.
esac
}
for f in .deep_crawl/sections/*.md; do
[ -f "$f" ] || continue
SECTION=$(basename "$f" .md)
WORDS=$(wc -w < "$f")
FACTS=$(grep -c '\[FACT' "$f" 2>/dev/null || echo 0)
[ "$WORDS" -eq 0 ] && continue
FLOOR=$(get_density_floor "$SECTION")
[ "$FLOOR" -eq 0 ] && continue # no density requirement for structural sections
DENSITY=$((FACTS * 100 / WORDS))
[ "$DENSITY" -lt "$FLOOR" ] && echo "FAIL: $SECTION — density ${DENSITY}/100w (floor: ${FLOOR})"
done
# Log section quality gate results
echo "## Section Quality Gate" >> "$CRAWL_LOG"
[ -n "$MISSING" ] && echo " Missing sections:$MISSING" >> "$CRAWL_LOG"
for f in .deep_crawl/sections/*.md; do
[ -f "$f" ] || continue
SECTION=$(basename "$f" .md)
[[ "$SECTION" == _* || "$SECTION" == S* || "$SECTION" == gotchas_from_* || "$SECTION" == header || "$SECTION" == footer || "$SECTION" == document_map ]] && continue
WORDS=$(wc -w < "$f"); FACTS=$(grep -c '\[FACT' "$f" 2>/dev/null || echo 0)
echo " $SECTION: ${WORDS}w, ${FACTS} FACT" >> "$CRAWL_LOG"
FLOOR=$(get_density_floor "$SECTION")
if [ "$FLOOR" -gt 0 ] && [ "$WORDS" -gt 0 ]; then
DENSITY=$((FACTS * 100 / WORDS))
if [ "$DENSITY" -ge "$FLOOR" ]; then
echo "$(date -Iseconds) GATE $SECTION PASS words=$WORDS facts=$FACTS density=${DENSITY}/100w" >> "$ORCH_LOG"
else
echo "$(date -Iseconds) GATE $SECTION FAIL words=$WORDS facts=$FACTS density=${DENSITY}/100w floor=${FLOOR}" >> "$ORCH_LOG"
fi
fi
done
echo "" >> "$CRAWL_LOG"
Thresholds are defined in .claude/skills/deep-crawl/configs/quality_gates.json under assembly_sections.density_tiers.
If missing sections found: spawn the appropriate assembly sub-agent (S1-S6 based on which section is missing).
If density fails: re-spawn the section agent with "Your section has {DENSITY} [FACT]/100w. Floor for {TIER} sections is {FLOOR}. Re-assemble with more citations from your findings input."
# === PLAYBOOK QUALITY GATE (mandatory after S6 — per playbook, not aggregate) ===
if [ -f .deep_crawl/sections/change_playbooks.md ]; then
# Split by ### headings into individual playbook files
mkdir -p .deep_crawl/_pb_check
csplit -z .deep_crawl/sections/change_playbooks.md \
'/^### /' '{*}' \
--prefix=.deep_crawl/_pb_check/pb_ \
--suffix-format='%03d.md' 2>/dev/null
PB_GATE=true
