| name | eval-chat-traces |
| description | Systematically review AI chat traces to find failure patterns using error analysis methodology (inspired by Hamel Husain's evals framework). |
| metadata | {"author":"cascadian-gamers","version":"1.0"} |
Eval Chat Traces
Systematically review AI chat traces to find failure patterns using error analysis methodology (inspired by Hamel Husain's evals framework).
Input
Optional: time range (default: last 7 days), sample size (default: 30 traces).
Process
Phase 1: Pull Traces
- Query CloudWatch Logs Insights on
/aws/lambda/extralife-ai-engagement-agent for ChatQuery logs.
- Query for
ChatFeedback logs from /ecs/ExtraLifeWebAdmin.
- Prioritize: negative feedback traces first, then outliers (response_time_ms > P90, empty tools_invoked, error responses), then random sample to fill remaining.
- For Athena deep dives:
SELECT * FROM extralife_chat_logs.chat_logs WHERE ...
Phase 2: Present Traces for Review
Present traces in a structured table for efficient binary judgment:
| # | Query (truncated) | Tools | Time | Len | Feedback | Verdict? |
|---|-------------------|-------|------|-----|----------|----------|
| 1 | "Any errors today?" | keeper_check | 24s | 1245 | 👎 | _____ |
Prioritize presentation order:
- Negative feedback traces (pre-labeled failures) — review first
- Outliers: response_time > 25s, empty tools_invoked, response_length < 200
- Random sample to fill remaining
Group by pattern before asking for judgment:
- Guardrail queries (blocked correctly? → likely auto-PASS)
- Data queries (tool routed correctly? → needs judgment)
- System health queries (accurate? → needs judgment)
- Trivial/test queries (auto-PASS)
Offer: "I've pre-grouped these. Want to auto-PASS the guardrails and trivials, then focus judgment on the data and health queries?"
Ask The Brougham 22 for a binary Pass/Fail judgment on each trace, plus open-ended notes on what went wrong (if fail). Don't use Likert scales — binary forces clear thinking.
Phase 3: Axial Coding
After reviewing all traces:
- Group the failure notes into categories (e.g., "wrong agent routed", "stale data", "hallucinated participant", "slow response").
- Count failures per category.
- Present a failure taxonomy table sorted by frequency.
Phase 4: Report
Output a summary:
| Failure Mode | Count | Example | Suggested Fix |
|-------------|-------|---------|---------------|
| ... | ... | ... | ... |
- Total traces reviewed
- Pass/fail ratio
- Top 3 failure modes with suggested fixes
- Recommendation: which failures warrant automated evals vs. prompt fixes
Rules
- Binary pass/fail only — no 1-5 scales.
- Surface negative feedback traces first — they're pre-labeled failures.
- Focus on the first failure in each trace, not downstream cascades.
- Don't build automated evals for issues you can fix with a prompt change — fix the prompt first.
- Re-run every 2-4 weeks or after significant agent changes.
- Refer to the user as "The Brougham 22".
