| name | hol-run-lab |
| description | Validate a HOL lab by simulating a participant. Walks through all 5 Acts (Explore, Fix, Spec, Build, Show & Tell) and produces a validation report with pass/fail, timing, and facilitator recommendations. |
HOL Lab Validator
You are simulating a workshop participant to validate that a HOL lab works end-to-end. Walk through all 5 Acts as if you were seeing the codebase for the first time. Be honest about what's confusing, what's too easy, and what doesn't work.
Important Rules
- Do NOT read
.hol/bugs/answers.md until AFTER you've attempted Act 2. The whole point is to validate that the bugs are discoverable without the answer key.
- Time your work mentally — note when each act starts and roughly how long it takes you. Human participants will take 2-3x longer.
- Be critical — if something is confusing, broken, or too easy, say so. A failed validation now saves 20 frustrated participants later.
- You cannot truly simulate a cold start. You've already read the codebase during planning. Focus your evaluation on whether a human newcomer would have enough context to orient themselves — not on your own experience.
Pass/Fail Rubric
Use these definitions consistently across all acts:
- FAIL — A blocker prevents completing the act. Examples: app won't start, seed data is broken/missing, bug is unfindable after 20 min of focused effort, feature is unbuildable in 30 min.
- PASS WITH NOTES — The act is completable but has meaningful friction. Examples: misleading error messages, confusing naming, an endpoint 500s on edge-case seed data, spec template is unclear on a section, bug is findable but the diagnosis path is indirect.
- PASS — The act works as designed with no significant friction.
Act 1 — Explore (Target: 10 min for humans)
Simulate a cold start. You've never seen this repo before.
- Read
CLAUDE.md — is it clear enough to get started?
- Install and run:
uv sync
uv run python seed.py
uv run pytest
uv run uvicorn main:app --reload
- Explore the codebase — map the architecture, understand the domain
- Trace one main data flow (e.g., "how does a booking get created?")
- Hit a few API endpoints to see what data exists
Evaluate:
Act 2 — Fix (Target: 20 min for humans)
Find and fix the planted bugs WITHOUT reading the answer keys.
To better simulate bounded knowledge, run the debugging in a subagent. Give it only the pytest output and instruct it to diagnose from symptoms — do not pre-load codebase context. This prevents your earlier exploration from short-circuiting the diagnosis path a human would follow.
- Run
pytest and capture the output
- Launch a subagent with the test output. The subagent should:
- Read the failing test to understand what it expects
- Trace the failure through the codebase (router → service → model → schema)
- Identify the root cause
- Apply the fix
- Write a regression test
- Run
pytest — all tests should pass after fixes
Evaluate:
After fixing: Now read .hol/bugs/answers.md and compare:
- Did you find the same root cause?
- Was your fix the same or equivalent?
- How long did it take vs the 20-min target?
If there is a stretch bug: Verify that leaving it unfixed does NOT block Acts 3-4. Revert the stretch bug fix, then confirm the recommended feature is still specifiable and buildable with the stretch bug present. If the stretch bug interferes with the feature exercise, flag it as a FAIL — stretch bugs must be safe to skip.
Act 3 — Spec (Target: 20 min for humans)
- Read
.hol/features/briefs.md (the participant-facing file — do NOT read proposals.md, which contains facilitator-only implementation details)
- Pick the recommended feature (or any feature)
- Read the feature brief (the 1-2 sentence description)
- Fill out
.hol/templates/feature-spec.md for the chosen feature
- Save the spec to
.hol/validation/feature-spec-filled.md
Evaluate:
Act 4 — Build (Target: 30 min for humans)
Implement the feature from your spec.
- Create the implementation: new/modified service functions, router endpoints, models if needed
- Write tests for the new feature
- Run the full test suite — everything should pass
- Verify the feature works via API
Evaluate:
Act 5 — Report
Generate a validation report at .hol/validation/report.md:
# Lab Validation Report — {App Name}
Date: {today}
## Summary
- Overall: PASS / FAIL / PASS WITH NOTES
- Lab is ready for: {audience} participants
## Act-by-Act Results
### Act 1 — Explore
- Status: PASS / FAIL
- Notes: {observations}
- Estimated human time: {X} min (target: 10 min)
### Act 2 — Fix
- Status: PASS / FAIL
- Bugs found: {N} / {N total}
- Diagnosis quality: {GOOD / FAIR / POOR — explain}
- Notes: {observations}
- Estimated human time: {X} min (target: 20 min)
### Act 3 — Spec
- Status: PASS / FAIL
- Notes: {observations}
- Estimated human time: {X} min (target: 20 min)
### Act 4 — Build
- Status: PASS / FAIL
- Feature built: {name}
- Complications: {any}
- Estimated human time: {X} min (target: 30 min)
## Issues Found
1. {issue — severity — recommendation}
2. ...
## Facilitator Recommendations
- {recommendation 1}
- {recommendation 2}
- ...
## Verdict
{Final assessment: is this lab ready for a live workshop?}
Undo all your changes after generating the report (the lab should be left in its "bugged" state for participants):
git checkout -- .
git clean -fd --exclude=.hol/validation/
The git checkout reverts tracked files. The git clean removes untracked files created during Act 4 (new services, tests, etc.) while preserving the validation report directory.
