triage-tester-feedback

name	triage-tester-feedback
description	Triage in-app tester feedback (thumbs / telemetry) and optional local logs. Diagnose bad responses through your product’s chat/RAG pipeline, apply minimal fixes, verify. Use when the user mentions feedback, flagged responses, tester issues, or bad answers.

Triage Tester Feedback

Workflow for handling flagged responses. Event names, tables, and scripts are product-specific — confirm against PRODUCT_CONTEXT and your bound codebase. This portable Method pack does not ship a real telemetry schema or internal ben_* modules.

Phase 1 — Gather

1. Query your feedback / telemetry store (common pattern: SQLite telemetry with an event type such as chat.feedback). Use [ORG_NAME_UPPER]_SQLITE_PATH or the DB URL your product documents. Example default path only: tmp/app.db — adopters override.
2. Optionally read logs/tester_feedback.jsonl (or equivalent) if your product defines it.
3. Group entries by sentiment, issue codes, tenant, or theme.
4. Present a short summary and ask which entries to investigate.

Phase 2 — Diagnose

Trace each flagged response through your pipeline layers in order. Stop where the failure first appears. Map each bullet to concrete files in the product repo (not this pack):

1. Session / history — wrong thread, cross-user bleed, stale context, tenant mix-ups.
2. Intent or routing — misrouted tools, wrong doc collections, incorrect “mode.”
3. Context extraction — missed or sticky entities from prior turns.
4. Retrieval — hybrid weights, filters, chunking, embedding / index drift.
5. Prompting / guardrails — over-refusal, confidence-band mistakes, safety-rule interaction bugs.
6. Model output — hallucinated identifiers, formatting, policy violations, contradictions.

Phase 3 — Communicate

Before disruptive restarts, follow your ops comms practice (chatops, email, or silent fix). Do not assume Discord scripts or channels exist.

Phase 4 — Fix

1. Make the smallest change that addresses the root layer.
2. Run your linters and tests (pytest, e2e, etc.) as required by the change.
3. Restart only the processes the change requires (API, worker, bot) per PRODUCT_CONTEXT.

Phase 5 — Verify and Close

1. Re-run the failing scenario; add a regression test when practical.
2. Log the finding in Engineering continuity / session notes per your org’s protocol.
3. If comms are expected, send an all-clear through your channel.

Pitfall patterns (generic)

Use as a checklist — adapt thresholds and filenames to your product:

Symptom	Often means
Wrong user’s context	Session keying or cache scope bug
Good docs not retrieved	Thresholds, filters, or stale index
Correct retrieval, wrong answer	Prompt / citation rules or model settings
Contradictory final text	Conflicting guardrails or post-processing