ralph-bug-hunt

name	ralph-bug-hunt
description	Autonomous hunt-and-fix loop for a single selectools module. Finds bugs, auto-applies fixes, writes regression tests, verifies with pytest. Outputs RALPH_RESULT sentinel on the last line so the orchestration script can detect convergence.
argument-hint	<module> [loops=N] e.g. "agent", "rag loops=5", "providers loops=1"

Ralph Bug Hunt

Autonomous bug hunt + auto-fix loop for module: $ARGUMENTS

If no module is given, default to "rag".

Loop Count

Parse the loops=N parameter from $ARGUMENTS if present. This controls how many hunt-fix-verify cycles to run before emitting the sentinel. Default: 1 (single pass).

• loops=1 (default) — one pass: hunt, fix, verify, emit sentinel
• loops=3 — three passes: after fixing, re-scan the same module for new bugs exposed by the fixes, fix those too, repeat until N passes or clean
• loops=0 — audit only: hunt and report but do NOT fix anything

When loops > 1, each pass re-reads all source files from scratch (fixes from pass 1 may expose new issues in pass 2). If a pass finds zero bugs, emit RALPH_RESULT: CLEAN immediately without running remaining passes.

Orchestration Script

The scripts/ralph_bug_hunt.sh script runs this skill in a convergence loop. Configure via environment variables:

MAX_ITER=10 REQUIRED_CLEAN=3 bash scripts/ralph_bug_hunt.sh        # all modules
MAX_ITER=5 REQUIRED_CLEAN=2 bash scripts/ralph_bug_hunt.sh rag     # single module
REQUIRED_CLEAN=1 bash scripts/ralph_bug_hunt.sh agent providers    # quick pass

What makes this different from /bug-hunt

1. Every finding is fixed immediately — no "ask user" step.
2. The last line of output is always a machine-parseable sentinel so the orchestration script (scripts/ralph_bug_hunt.sh) can detect convergence.

---

Step 0 — Parse Arguments

Parse $ARGUMENTS to extract:

• module: the first word (e.g. "rag", "agent")
• loops: if loops=N appears, extract N (integer). Default: 1.

Example: rag loops=3 → module="rag", loops=3 Example: agent → module="agent", loops=1 Example: providers loops=0 → module="providers", loops=0 (audit only, no fixes)

If loops=0, skip Steps 3-4 entirely. Only run Steps 1-2 (scan + report) and Step 5 (sentinel).

If loops > 1, wrap Steps 2-4 in a loop. After each pass, if zero findings, break early.

---

Step 1 — Scope Resolution

Map the module argument to its source paths:

Module	Source paths
agent	src/selectools/agent/
providers	src/selectools/providers/
tools	src/selectools/tools/, src/selectools/toolbox/
rag	src/selectools/rag/
memory	src/selectools/memory.py, src/selectools/entity_memory.py, src/selectools/knowledge*.py, src/selectools/sessions.py
evals	src/selectools/evals/
security	src/selectools/guardrails/, src/selectools/audit.py, src/selectools/security.py, src/selectools/coherence.py, src/selectools/policy.py

---

Step 2 — Bug Hunt Analysis

Read all source files for the module. Hunt for bugs in these 6 categories:

Category 1 — Correctness

• Type mismatches: function signatures don't match callers
• Async/sync inconsistency: arun()/astream() missing features that run() has
• None handling: using fields without or "" / or [] guards
• Race conditions: shared mutable state without locks in concurrent paths
• Resource leaks: executors, file handles, DB connections created per-call

Category 2 — API Contract

• Provider protocol: stream()/astream() not passing tools parameter
• ToolCall stringification in streaming paths
• Observer events missing run_id or not firing in all execution paths
• StepType using string literals instead of StepType.ENUM_NAME

Category 3 — Security

• SQL injection: raw f-strings in SQLite queries
• Path traversal: user-controlled paths in file operations (test with ../../etc/passwd)
• Prompt injection in evaluators: case.input/case.reference interpolated into LLM judge prompts without fencing
• IDOR: session IDs guessable or not validated

Category 4 — Memory & Performance

• Unbounded growth: lists/dicts that grow without limits
• Non-atomic file writes: path.write_text() without .tmp → os.replace() pattern
• Blocking sync I/O in async paths
• ThreadPoolExecutor() created per-call instead of module singleton

Category 5 — Edge Cases

• Empty inputs: 0 messages, 0 tools, empty strings, zero-length lists
• Zero/negative numeric parameters: max_iterations=0, budget=0, top_k=0
• Provider returns empty: empty string content, None tool_calls
• Concurrent tool execution modifying shared state

Category 6 — Documentation Drift

• Docstrings describing behavior the code doesn't implement
• Type hints claiming Optional but value is never actually None (or vice versa)

---

Step 3 — Fix Each Finding

For each bug found (Critical first, then High, Medium, Low):

1. Read the affected file in full.
2. Apply the fix using the Edit tool.
3. Write a regression test in the appropriate test file:
   • tests/<module>/test_<bug_description>_regression.py, or append to an existing regression file like tests/rag/test_rag_regression_phase3.py.
4. Verify the fix by running pytest on just that test file:

   pytest <test_file> -x -q
   

   • If the test passes → record as FIXED.
   • If the test fails → revert the code change (restore the original with Edit), leave the test commented out with a # UNFIXED: prefix, and record as UNFIXED.

---

Step 4 — Suite Verification

After all individual fixes are applied and verified, run the full suite (minus e2e):

pytest tests/ -k "not e2e" -x -q

If the suite fails:

• Identify which fix broke the suite.
• Revert that specific fix.
• Re-run the suite and confirm it passes.
• Reclassify that finding as UNFIXED.

---

Step 5 — Emit Sentinel (MANDATORY — this must be the LAST line of output)

Count findings and fixed vs unfixed:

If zero findings OR all findings were UNFIXED (meaning no net code change that broke anything):

RALPH_RESULT: CLEAN

If any findings were FIXED:

RALPH_RESULT: FOUND <N> (CRITICAL:<c> HIGH:<h> MEDIUM:<m> LOW:<l>) FIXED:<f>

Where:

• <N> = total findings
• <c>, <h>, <m>, <l> = count per severity
• <f> = number successfully fixed (test passed + suite passed)

IMPORTANT: The sentinel line must be the very last line printed. Do not add any text after it. The orchestration script uses tail -1 to detect convergence.

---

False Positive Handling

Before fixing, check if an existing test explicitly asserts the current behavior. A passing test that validates the "buggy" code is evidence the finding may be wrong — investigate before changing.

Why These Categories Catch What Tests Miss

• Thread-safety: tests run sequentially with mocks — races are invisible
• Injection/path traversal: tests use normal inputs — adversarial paths never tried
• Non-atomic writes: tests don't simulate process crashes
• None content: mocks return valid data — None content never exercised