| name | apex-debug |
| description | Use when the operator reports that APEX is misbehaving and wants it diagnosed — Slack bot silent/wrong, MCP/tool call failing, web/API 5xx, or orchestrator misrouting. Triggers on /apex-debug, "apex bozuk", "apex debug et", "bot cevap vermiyor", "tool patlıyor", "dashboard hata veriyor", "orchestrator yanlış routing", "neden çalışmıyor". Runs a read-only triage → evidence → root-cause → fix-in-worktree → architect-verify flow and STOPS at a verified fix proposal (no commit/push/deploy). |
APEX Debug Agent
Diagnose an APEX malfunction end-to-end and hand the operator a verified fix
proposal — root cause, a worktree diff an architect signed off on, a deploy plan, and
a smoke-test plan. Never commit, push, or deploy. The operator approves what happens
next.
Design spec: docs/superpowers/specs/2026-06-11-apex-debug-agent-design.md
The brain: docs/debug/apex-diagnostic-kb.md (read this — it holds the symptom→evidence→owner map).
Hard boundaries
- No commit / push / deploy. The run ends at the report (§6). Remote git and deploy
are operator-approved actions (Golden Rule §6, AWS HARD RULE).
- All remote evidence-gathering is READ-ONLY. No mutating command in qa/test/prod
without explicit operator approval. No
0.0.0.0/0, no delete, no public exposure.
- Honest verification (Golden Rule §2). The full live golden-path smoke needs a
deploy → operator-gated. Verify what you can in the worktree (architect review +
targeted/unit tests) and state explicitly what could NOT be verified without deploying.
- Don't reinvent. Use the runbooks, SYSTEM.md, lessons, and memory the KB points to.
SOP
You MUST create a TodoWrite item per step and work them in order.
0. Intake
Capture the symptom and the target environment. If the operator did not say
env=dev|qa|test|prod, ask before gathering any evidence — the adapter and access
method depend on it.
1. Triage
Read docs/debug/apex-diagnostic-kb.md. Match the symptom to one or more of the 4
classes (Slack / MCP-tool / Web-API / Orchestrator). Select the environment adapter:
| env | Logs | DB | API / UI | Slack |
|---|
| dev | docker compose logs | localhost:5433 (apex/apex_dev) | :8000 / https://localhost | token + channel in hand |
| qa | ssh ec2_console → kubectl -n apex-ai logs | kubectl exec psql | apex-test.getirfinance.com | isolated (no LDAP/DC) |
| test | ssh svmaisgentst001 (sudo) | in-place | in-place | — |
| prod | operator grants access; read-only, no command without approval | — | — | — |
2. Evidence (subagent, read-only)
Dispatch issue-analyzer (or Explore for pure location) — do NOT gather in the
main loop. Give it a self-contained prompt: the symptom, the env + exact access method,
the KB's "where to look" list for the matched class(es), and an instruction to stay
read-only and return ranked findings with file:line / log evidence. Run classes in
parallel (one subagent each) when the symptom spans more than one. Pipe large logs
through ctx_execute so raw bytes never enter context.
3. Root cause
Synthesize the evidence into a ranked root-cause hypothesis with file:line proof.
Confirm code is actually at fault before step 4 — many APEX failures are config /
credential state (expired OAuth, disabled flag, un-recreated container), not code. If
it's config, skip to a report telling the operator the exact reconnect/flag/recreate.
4. Fix (owning developer agent, worktree)
Dispatch the symptom class's owning agent (KB "Owning agent" field) with
isolation: worktree: orchestrator-developer / inference-developer /
mcp-developer / rag-developer / devops-engineer. Self-contained prompt: the root
cause, the constraint to make a minimal change, and the protected-features list (don't
break shipped work).
5. Verify (architect gate)
Dispatch software-architect to review the worktree diff: correctness, harmony with
existing architecture, and that it breaks no protected feature. Run targeted/unit tests
in the worktree. Architect renders GO / NO-GO. On NO-GO, loop back to step 4 with the
architect's objection.
6. Report — STOP
Deliver, in plain language (Golden Rule §4):
- Root cause — with
file:line evidence.
- Verified diff — the worktree change + architect's GO.
- Deploy plan — exact steps for this env (e.g.
docker compose -p self_apex up -d --no-deps --force-recreate <svc>).
- Smoke-test plan — the golden path to run after apply (chain into
apex-live-tests if the operator approves applying).
- Not verified without deploy — explicit list of what the worktree review could not prove.
Then stop and let the operator decide. Do not commit, push, or deploy.
Future expansion (do not build now — see spec §8)
- B (Workflow): parallel investigators + adversarial root-cause verification for complex multi-symptom cases.
- C (Skill→Workflow): this skill launches the workflow for complex cases.
- Hybrid → APEX specialist:
docs/debug/apex-diagnostic-kb.md becomes the system-prompt knowledge of an APEX orchestrator diagnostics specialist; same brain, APEX's own tools as the hands.
