| name | rumil-find-confusion |
| description | Scan recent rumil calls for signs of model confusion. Default mode uses fast, free heuristics (error events, non-complete status, exchange errors, cost outliers, thin output). Pass --deep for an LLM-based structured verdict on the top heuristic candidates, cached in scan log. Pass --structural <question_id> for graph health checks on a question's subtree (barren questions, orphans, load-bearing fragile claims). Use when the user wants to review recent runs for quality, triage a batch of calls, or find specific traces worth inspecting. |
| allowed-tools | Bash |
| argument-hint | [--limit N] [--deep [--deep-limit K] [--model ...]] [--force-rescan] [--structural <question_id>] |
rumil-find-confusion
Triages recent calls in the active workspace, surfacing ones that look
off. Three modes:
Heuristic (default) — fast, deterministic, no LLM cost (only
local Supabase reads + arithmetic). Scores each recent call by:
- hard signals: non-complete status, trace error events, exchange errors
- soft signals: cost outliers (> 3× median), thin output relative to
input, multiple warnings
Structural (--structural <question_id>) — graph health checks on
a question's subtree. Detects barren questions, unjudged questions,
orphaned claims, load-bearing fragile claims, dead-end decompositions,
and chained supersession. Pure DB queries, no LLM cost.
Deep (--deep) — for the top heuristic candidates, runs a meta LLM
call with a shared system prompt (designed for prompt-cache reuse
across many scans) and a structured ConfusionVerdict schema.
Costs per-scan LLM tokens, roughly:
claude-haiku-4-5-20251001 — ~$0.005 per traceclaude-sonnet-4-6 (default) — ~$0.02-0.05 per traceclaude-opus-4-7 — ~$0.10-0.20 per trace
Exact cost depends on trace size (bigger traces = more tokens in the
user message). Verdicts are cached in
.claude/state/rumil-scan-log.json, so re-running the skill on a
trace already in the log doesn't re-pay. Force a re-scan with
--force-rescan.
Override model with --model <id> (e.g. haiku for cheap bulk scans).
Invocation
setopt no_glob 2>/dev/null; set -f; PYTHONPATH=.claude/lib uv run python -m rumil_skills.find_confusion $ARGUMENTS
After it runs
Heuristic output is a ranked list: short ID, call type, signals, score.
Deep output adds a structured verdict per call — primary symptom,
severity, evidence quotes, suggested action (inspect, redispatch,
edit_prompt:<file>, ignore).
For any call that looks worth investigating, the natural next step is
/rumil-trace <call_id> to read the full exchanges, or
/rumil-review <question_id> if the issue seems question-wide.
