| name | assistant-test |
| description | Test the personal assistant's routing accuracy and response quality. Run with /assistant-test to execute all cases, /assistant-test add to add new cases, /assistant-test <id> for a single case. |
Assistant Test Suite
Test routing accuracy and response quality across all skills.
Commands
| Command | What it does |
|---|
/assistant-test | Run all test cases, report pass/fail + token stats |
/assistant-test add | Add a new test case interactively |
/assistant-test <id> | Run a single test by ID |
/assistant-test routing | Run only routing tests (fast, no actual execution) |
/assistant-test quality | Run full quality tests (slower, checks response content) |
/assistant-test stats | Show token usage summary from last run |
Test File
Tests live in <vault>/persona/tests/cases.md (editable in Obsidian as markdown tables).
Cases are grouped under ## <Section> headings (e.g. ## Game Time, ## KB - Immigration). Each section contains a markdown table with columns: ID | Input | Skill | Expect Contains | Tags.
Parse the tables to extract test cases. Expect Contains and Tags are comma-separated; an empty Expect Contains cell means skip the content check.
Example:
## Game Time
| ID | Input | Skill | Expect Contains | Tags |
|----|--------------|-----------|-----------------|--------------------|
| 1 | <name>工作1小时 | game-time | <name>, 工作余额 | routing, game-time |
Fields
| Field | Description |
|---|
id | Unique test ID |
input | Simulated Telegram message |
expect_skill | Which skill should handle this: game-time, kb, calendar, gmail, websearch, memory, task, direct |
expect_contains | Key strings that must appear in the response (empty = skip content check) |
tags | Categories for filtering |
Running Tests
Step 1: Load
Read <vault>/persona/tests/cases.md. Walk every ## section; for each, parse the markdown table into rows. Skip rows where the ID column is missing or non-numeric (header / divider rows).
Step 2: For Each Test
Routing test (fast):
- Read the
input message
- Based on CLAUDE.md's skill catalog and available skills, decide which skill would handle it
- Compare against
expect_skill
- Record: PASS if correct, FAIL if wrong skill chosen
Quality test (full):
- Do the routing test above
- Actually execute the skill (read kb articles, check game-time data, etc.)
- Generate the response (but do NOT send to Telegram)
- Check
expect_contains - all listed strings must appear in the response
- Record: PASS/FAIL with evidence
Token tracking:
- Before each test, note the conversation position
- After each test, estimate tokens used:
- Input: count characters of all files read × 0.3 (rough char->token ratio)
- Output: count characters of response × 0.3
- Record in results
Step 3: Report
Print a human-readable summary: routing pass/fail list, quality pass/fail list, totals, token estimate, and a per-skill breakdown. Exact layout: see references/output-formats.md § "Step 3: Report format".
Step 4: Save Results
Write a results JSON to <vault>/persona/tests/YYYY-MM-DD-HHMMSS.json (per-test results + a token_summary aggregated by skill). Full schema: see references/output-formats.md § "Step 4: Save Results".
Adding Tests
/test add flow:
- Ask: "What message would you test?"
- Ask: "Which skill should handle it?" (suggest based on message content)
- Run the message through the assistant (dry run, no Telegram send)
- Show response, ask: "Are these the key facts to check?"
- Append a new row to the matching
## <Section> table in <vault>/persona/tests/cases.md. Pick the next free ID (max(existing) + 1). If no section fits, add a new ## <Section> block with a fresh table.
Dry Run Mode
All tests run in dry run mode:
- Skills are executed but responses are NOT sent to Telegram
- Game-time balances are NOT actually updated
- KB articles are read but no files are modified
- Calendar/Gmail MCP tools are called read-only (list/search, not create/modify)
- Results are captured for verification only
Comparing Runs
/test stats reads the two most recent results files and shows:
- Tests that flipped (pass->fail or fail->pass)
- Token usage changes
- New failures to investigate
Integration with KB Tests
KB content-accuracy tests are filed in cases.md under the ## KB - <topic> sections (tagged kb). This unified test suite covers the full assistant pipeline: routing -> skill execution -> response quality.
/assistant-test routing = "does the assistant pick the right skill?"/assistant-test quality = "does the response contain the key facts?"
