| name | test-bot |
| description | Agentic end-to-end testing of the bot via terminal-chat. Spins up docker compose, pipes test conversations, validates responses, and tears down. Use after implementing a feature to verify it works live. |
| argument-hint | ["scenario"] |
| allowed-tools | Bash, Read, Grep, Glob |
Agentic Bot Testing
Run end-to-end agentic tests against the live bot using terminal-chat with docker compose infrastructure.
Setup
- Start infrastructure (skip if already running):
docker compose up -d postgres dragonfly nats
- Wait for postgres to be healthy:
until docker compose ps postgres --format '{{.Status}}' | grep -q healthy; do sleep 1; done
- Export environment variables (always enable dev mode for testing):
export $(grep -v '^#' .env | grep -v '^$' | xargs) && export LEARN_DEV_MODE=true
Test Execution
Run conversations through terminal-chat with --memory flag (in-memory state, no DB dependency for the store):
printf '<test_input>\n' | timeout 30 go run ./cmd/terminal-chat/ --memory 2>&1
Each test pipes one or more messages and checks the output for expected patterns.
Important Lessons
- Always start with
/clear or /dev-reset to ensure clean state when testing mastery/progress features
--memory flag means each go run invocation starts fresh — mastery only accumulates within a single piped session- Mastery grading is async — runs in a goroutine after AI responds. In piped sessions, there may not be enough time for the goroutine to complete before the next message is processed. For mastery/progress tests, include enough teaching turns (6+) in a single piped session
/dev-reset (requires LEARN_DEV_MODE=true) fully clears mastery, XP, streaks, goals, and profile — use this to verify clean state- Unlock notifications are drained at the top of
ProcessMessage — they appear on the message after mastery crosses the threshold, on any message type (commands or chat)
Scenario: $ARGUMENTS
If $ARGUMENTS is provided, test that specific feature/command. Otherwise, run the standard smoke test suite below.
Standard Smoke Test Suite
Run these scenarios in order. For each one, pipe the input and verify the output contains expected strings.
1. Command: /learn (no args → usage)
printf '/learn\n' | timeout 15 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: output contains /learn (usage hint)
2. Command: /learn with valid topic
printf '/learn persamaan linear\n' | timeout 15 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: output contains topic name (e.g., "Persamaan Linear" or "Linear Equations")
3. Command: /learn with invalid topic
printf '/learn quantum physics\n' | timeout 15 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: output contains "tidak dijumpai" or "not found"
4. Command: /progress
printf '/progress\n' | timeout 15 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: output contains "Progress" or "XP", no error/panic
5. Command: /goal
printf '/goal kuasai algebra\n' | timeout 30 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: output mentions goal or topic, no error/panic
6. Free-form teaching message
printf 'Ajar saya tentang persamaan linear\n' | timeout 30 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: AI responds with teaching content (non-empty response after "P&AI>")
7. Unknown command
printf '/foobar\n' | timeout 15 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: output contains "tidak diketahui" or "Unknown command"
8. Multi-turn teaching + mastery accumulation
printf '/learn persamaan linear\nPersamaan linear x + 3 = 7, tolak 3, x = 4\n2x + 4 = 10, tolak 4, 2x = 6, bahagi 2, x = 3\n3x - 9 = 0, tambah 9, 3x = 9, bahagi 3, x = 3\n5x + 10 = 25, tolak 10, 5x = 15, bahagi 5, x = 3\n4x - 8 = 12, tambah 8, 4x = 20, bahagi 4, x = 5\n/progress\n' | timeout 180 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: /progress shows F1-06 with mastery > 0%, XP > 0, streak 1 day
9. /dev-reset hidden without dev mode
LEARN_DEV_MODE=false go run ./cmd/terminal-chat/ --memory <<< '/dev-reset' 2>&1
Note: This won't work with env prefix; use export LEARN_DEV_MODE=false before running.
Expect: output contains "tidak diketahui" or "Unknown command"
10. /dev-reset clears everything
printf '/learn persamaan linear\nPersamaan linear x+3=7, x=4\n2x+4=10, x=3\n3x-9=0, x=3\n5x+10=25, x=3\n4x-8=12, x=5\n/progress\n/dev-reset\n/progress\n' | timeout 180 go run ./cmd/terminal-chat/ --memory 2>&1
Expect: First /progress shows mastery/XP > 0. After /dev-reset, second /progress shows XP: 0 and no mastery.
Validation
For each scenario:
- Run the command
- Check exit code (should be 0)
- Check output contains expected string (case-insensitive)
- Report PASS/FAIL with the actual output on failure
Teardown
After all tests, stop infrastructure:
docker compose stop postgres dragonfly nats
Report Format
Agentic Test Results
====================
1. /learn (no args) PASS
2. /learn valid topic PASS
3. /learn invalid topic PASS
4. /progress PASS
5. /goal PASS
6. Teaching message PASS
7. Unknown command PASS
8. Multi-turn mastery PASS
9. /dev-reset hidden PASS
10. /dev-reset clears all PASS
Result: 10/10 passed
If any test fails, show the expected vs actual output.
