skill-tester

name

skill-tester

description

Test an interactive lesson/course (or any "instructions to an AI" skill) by self-play. An agent plays BOTH the instructor following the lesson script AND a calibrated student persona, producing full turn-by-turn transcripts of every lesson, then publishes the raw transcripts to a single static page. Use when asked to "run lesson transcripts", "test the course end to end", "self-play the lessons", "publish raw test transcripts", "walk a synthetic student through every lesson", or to QA an interactive-instruction skill by actually running it rather than just reviewing findings. Distinct from dogfood and adversarial-ux-test (web-app browser QA) and synthetic-userstudies (findings plus a few cherry-picked transcripts). This one captures the COMPLETE run of every lesson and ships them all raw.

skill-tester — self-play lesson transcripts to a static page

Test an interactive course by RUNNING it, not reviewing it. Each lesson file is "instructions to an AI on how to teach." A faithful test executes those instructions against a simulated student and captures the whole conversation. Publish every transcript raw so a human can read exactly how each lesson plays out and where it snags.

This is different from the QA skills you may already have:

• dogfood / adversarial-ux-test — browser automation against a deployed web app.
• synthetic-userstudies — produces ranked FINDINGS plus a couple of highest-signal transcripts. NOT full raw runs of every lesson.
• skill-tester — full turn-by-turn transcript of EVERY lesson, all published raw.

When to run

After a lesson rewrite or new lessons, when the owner wants to SEE the lessons run rather than read a findings memo. Run against the freshest source (pull the default branch, use a worktree; local checkouts go stale).

Step 1 — Grab the freshest source

Pull the repo, add a worktree on the default branch, confirm the lesson count matches the latest merged work. Read any CLAUDE.md / AGENTS.md / LESSON-FORMAT.md / skill spec so the instructor half follows the real teaching rules (e.g. "one step per message, STOP and wait, DO the demo live, keep messages short").

Step 2 — Self-play dispatch (parallel, read-only)

Group lessons (e.g. by section, ~4 each) and dispatch one read-only subagent per group, toolsets limited to terminal + file, instructed NOT to edit anything except its own output files. Each subagent, per lesson, writes a VERBATIM transcript where it plays both roles:

• INSTRUCTOR: follow the lesson file exactly — one step per message, run/narrate the demo, use the lesson's actual prompts and visuals, STOP after each step.
• STUDENT: a CALIBRATED persona (see below). Reacts in good faith, sometimes hits a snag, asks a clarifier, or pastes a plausible result.

Per-lesson transcript format: a # Lesson N: title heading, a one-line claimed magic moment, the turn-by-turn body with **INSTRUCTOR:** / **STUDENT:** labels covering the WHOLE lesson, then a ### Where it snagged tail (1-3 real friction bullets, or "clean run"). Label every file SYNTHETIC.

Step 3 — Calibrate the persona (the load-bearing decision)

The most fragile beginner persona is a FALSE FLOOR — it over-drives fixes and patronizes the real audience. State the REAL audience explicitly before running (for an AI/dev course it's typically a tool-literate professional: knows what a system prompt is, can install an app and run a setup command). Assume competence. Keep skill-independent findings (silent-failure bugs, missing-prerequisite fallbacks, overgeneralizations that lose a skeptic); discount false-floor-only complaints.

Step 4 — Publish raw

Build a single static page hosting ALL transcripts raw, collapsible per lesson, grouped by section, with a SYNTHETIC warning header. See references/build-transcripts-page.md for the build script and styling. A typical static-host deploy:

python3 build_transcripts.py        # writes index.html
cp index.html 200.html              # SPA fallback for hosts that need it
# deploy index.html + 200.html to any static host

Verify the live page with a real rendered screenshot, not just a DOM grep — static CDNs can return a transient error on a cold first hit, retry after a few seconds.

Pitfalls

• Local repo checkout is usually STALE. Always work from the remote default branch in a worktree.
• Read-only subagents get NO edit toolset beyond writing their own transcript files.
• Long URLs / code blocks force horizontal scroll on mobile — the build script sets white-space:pre-wrap; word-break:break-word and overflow-wrap:anywhere so they wrap.
• Don't cherry-pick. The whole point vs the findings-style test is that EVERY lesson ships raw, snags and all.