| name | triage-tester-feedback |
| description | Triage in-app tester feedback (thumbs / telemetry) and optional local logs. Diagnose bad responses through your product’s chat/RAG pipeline, apply minimal fixes, verify. Use when the user mentions feedback, flagged responses, tester issues, or bad answers. |
Triage Tester Feedback
Workflow for handling flagged responses. Event names, tables, and scripts are product-specific — confirm against PRODUCT_CONTEXT and your bound codebase. This portable Method pack does not ship a real telemetry schema or internal ben_* modules.
Phase 1 — Gather
- Query your feedback / telemetry store (common pattern: SQLite
telemetry with an event type such as chat.feedback). Use [ORG_NAME_UPPER]_SQLITE_PATH or the DB URL your product documents. Example default path only: tmp/app.db — adopters override.
- Optionally read
logs/tester_feedback.jsonl (or equivalent) if your product defines it.
- Group entries by sentiment, issue codes, tenant, or theme.
- Present a short summary and ask which entries to investigate.
Phase 2 — Diagnose
Trace each flagged response through your pipeline layers in order. Stop where the failure first appears. Map each bullet to concrete files in the product repo (not this pack):
- Session / history — wrong thread, cross-user bleed, stale context, tenant mix-ups.
- Intent or routing — misrouted tools, wrong doc collections, incorrect “mode.”
- Context extraction — missed or sticky entities from prior turns.
- Retrieval — hybrid weights, filters, chunking, embedding / index drift.
- Prompting / guardrails — over-refusal, confidence-band mistakes, safety-rule interaction bugs.
- Model output — hallucinated identifiers, formatting, policy violations, contradictions.
Phase 3 — Communicate
Before disruptive restarts, follow your ops comms practice (chatops, email, or silent fix). Do not assume Discord scripts or channels exist.
Phase 4 — Fix
- Make the smallest change that addresses the root layer.
- Run your linters and tests (
pytest, e2e, etc.) as required by the change.
- Restart only the processes the change requires (API, worker, bot) per PRODUCT_CONTEXT.
Phase 5 — Verify and Close
- Re-run the failing scenario; add a regression test when practical.
- Log the finding in Engineering continuity / session notes per your org’s protocol.
- If comms are expected, send an all-clear through your channel.
Pitfall patterns (generic)
Use as a checklist — adapt thresholds and filenames to your product:
| Symptom | Often means |
|---|
| Wrong user’s context | Session keying or cache scope bug |
| Good docs not retrieved | Thresholds, filters, or stale index |
| Correct retrieval, wrong answer | Prompt / citation rules or model settings |
| Contradictory final text | Conflicting guardrails or post-processing |
