assistant-test

name	assistant-test
description	Test the personal assistant's routing accuracy and response quality. Run with /assistant-test to execute all cases, /assistant-test add to add new cases, /assistant-test <id> for a single case.

Assistant Test Suite

Test routing accuracy and response quality across all skills.

Commands

Command	What it does
/assistant-test	Run all test cases, report pass/fail + token stats
/assistant-test add	Add a new test case interactively
/assistant-test <id>	Run a single test by ID
/assistant-test routing	Run only routing tests (fast, no actual execution)
/assistant-test quality	Run full quality tests (slower, checks response content)
/assistant-test stats	Show token usage summary from last run

Test File

Tests live in <vault>/persona/tests/cases.md (editable in Obsidian as markdown tables).

Cases are grouped under ## <Section> headings (e.g. ## Game Time, ## KB - Immigration). Each section contains a markdown table with columns: ID | Input | Skill | Expect Contains | Tags.

Parse the tables to extract test cases. Expect Contains and Tags are comma-separated; an empty Expect Contains cell means skip the content check.

Example:

## Game Time

| ID | Input        | Skill     | Expect Contains | Tags               |
|----|--------------|-----------|-----------------|--------------------|
| 1  | <name>工作1小时 | game-time | <name>, 工作余额  | routing, game-time |

Fields

Field	Description
id	Unique test ID
input	Simulated Telegram message
expect_skill	Which skill should handle this: game-time, kb, calendar, gmail, websearch, memory, task, direct
expect_contains	Key strings that must appear in the response (empty = skip content check)
tags	Categories for filtering

Running Tests

Step 1: Load

Read <vault>/persona/tests/cases.md. Walk every ## section; for each, parse the markdown table into rows. Skip rows where the ID column is missing or non-numeric (header / divider rows).

Step 2: For Each Test

Routing test (fast):

1. Read the input message
2. Based on CLAUDE.md's skill catalog and available skills, decide which skill would handle it
3. Compare against expect_skill
4. Record: PASS if correct, FAIL if wrong skill chosen

Quality test (full):

1. Do the routing test above
2. Actually execute the skill (read kb articles, check game-time data, etc.)
3. Generate the response (but do NOT send to Telegram)
4. Check expect_contains - all listed strings must appear in the response
5. Record: PASS/FAIL with evidence

Token tracking:

• Before each test, note the conversation position
• After each test, estimate tokens used:
  • Input: count characters of all files read × 0.3 (rough char->token ratio)
  • Output: count characters of response × 0.3
  • Record in results

Step 3: Report

Print a human-readable summary: routing pass/fail list, quality pass/fail list, totals, token estimate, and a per-skill breakdown. Exact layout: see references/output-formats.md § "Step 3: Report format".

Step 4: Save Results

Write a results JSON to <vault>/persona/tests/YYYY-MM-DD-HHMMSS.json (per-test results + a token_summary aggregated by skill). Full schema: see references/output-formats.md § "Step 4: Save Results".

Adding Tests

/test add flow:

1. Ask: "What message would you test?"
2. Ask: "Which skill should handle it?" (suggest based on message content)
3. Run the message through the assistant (dry run, no Telegram send)
4. Show response, ask: "Are these the key facts to check?"
5. Append a new row to the matching ## <Section> table in <vault>/persona/tests/cases.md. Pick the next free ID (max(existing) + 1). If no section fits, add a new ## <Section> block with a fresh table.

Dry Run Mode

All tests run in dry run mode:

• Skills are executed but responses are NOT sent to Telegram
• Game-time balances are NOT actually updated
• KB articles are read but no files are modified
• Calendar/Gmail MCP tools are called read-only (list/search, not create/modify)
• Results are captured for verification only

Comparing Runs

/test stats reads the two most recent results files and shows:

• Tests that flipped (pass->fail or fail->pass)
• Token usage changes
• New failures to investigate

Integration with KB Tests

KB content-accuracy tests are filed in cases.md under the ## KB - <topic> sections (tagged kb). This unified test suite covers the full assistant pipeline: routing -> skill execution -> response quality.

• /assistant-test routing = "does the assistant pick the right skill?"
• /assistant-test quality = "does the response contain the key facts?"