| name | product-journey-qa |
| description | Run or refine the Kitsoki product-journey QA pipeline: generate 10-GitHub-repo natural-use matrices, create persona/scenario run bundles, drive or hand off visual/Kitsoki MCP evidence capture, run no-LLM replay/dogfood gates, review/validate bundles, and produce Slidey decks with playback evidence. Use when asked to dogfood onboarding, bugfix, PRD/design, feature implementation, product-discovery, product-bug, personas, or the reusable product-journey QA agent/story. |
Product Journey QA
Use this skill when the task is to evaluate Kitsoki as a product through
repeatable persona journeys, not when merely editing one unrelated story room.
The durable surfaces are:
- Runner:
tools/product-journey/run.py
- Story wrapper:
stories/product-journey-qa/app.yaml
- Driver agent:
.agents/agents/product-journey-qa-driver.md
- Catalogs:
tools/product-journey/personas.json,
tools/product-journey/scenarios.json,
tools/product-journey/github-targets.json
- Review output:
.artifacts/product-journey/**/deck.slidey.json,
review.json, media-manifest.json, scenario-outcomes.md,
driver-journal.md, driver-handoff.md
Operating Rules
- Automated tests and repeatable gates must not call a real LLM. Use flow
fixtures, cassettes, demo evidence, or replay artifacts.
- Live/model work is only for explicitly authorized exploratory runs. If a
scenario needs live authorization and none was given, record a blocker instead
of faking evidence.
- Proof evidence sources are
local, retained, external, and cassette.
demo evidence exercises aggregation only; it is not product proof.
- Preserve the persona lens. A core maintainer, dependency debugger,
docs-minded contributor, IDE-first engineer, and hobbyist contributor should
start from different surfaces, ask different first questions, and weigh
evidence differently.
- Every scenario attempt needs a driver journal event, even when it ends in a
blocker.
- A bundle is not discussion-ready until
--review-run and --validate-run
have run and the deck has playback media or an explicit playback blocker.
No-LLM Gates
Start with the cheap gates before live capture:
python3 tools/product-journey/run.py --validate-corpus --json-output
python3 tools/product-journey/run.py --driver-replay-sweep --seed demo --json-output
GOCACHE=/private/tmp/kitsoki-gocache go run ./cmd/kitsoki test flows stories/product-journey-qa/app.yaml
Use --driver-replay-smoke --smoke-scenario <scenario-id> when narrowing a
single scenario. Use --dogfood-smoke when checking matrix-to-rollup artifact
composition.
Matrix Workflow
For the 10 popular GitHub repositories with at least 100 open bugs:
python3 tools/product-journey/run.py --refresh-github-targets --seed <seed>
python3 tools/product-journey/run.py --emit-matrix --seed <seed> \
--target-proof-file .artifacts/product-journey/target-proofs/<proof-id>
python3 tools/product-journey/run.py --validate-matrix \
--matrix-dir .artifacts/product-journey/matrices/<matrix-id> \
--strict-target-proof
Use --matrix-personas all when each target should be paired with every
persona. Use normal --validate-matrix for draft matrices; use
--strict-target-proof before a live scored sweep so missing refreshed GitHub
bug/star/license proof is an error. The matrix is an assignment plan, not
evidence that Kitsoki worked.
Run Bundle Workflow
Create one bundle from a matrix assignment or direct target:
python3 tools/product-journey/run.py --emit-run \
--project <target-id> --persona <persona-id> --seed <seed>
Then hand it to the reusable driver:
- Read
agent-brief.md, driver-plan.md, and driver-handoff.md.
- Use
.agents/agents/product-journey-qa-driver.md for live/cassette MCP
capture.
- Attach evidence with
--attach-evidence or the story attach intent;
loaded runs also expose last_result.next_driver_attach_command.
- Record findings with
--record-finding; record honest blockers with
--record-blocker or last_result.next_driver_blocker_command.
- Record each attempt with
--record-driver-event or the story driver_event
intent.
- Run:
python3 tools/product-journey/run.py --review-run --run-dir <run-dir>
python3 tools/product-journey/run.py --validate-run --run-dir <run-dir>
Review deck.slidey.json for the narrative, Playback evidence scenes,
Proof gates, Persona lens, and Driver contract scenes.
Story Surface
When driving through Kitsoki itself, open stories/product-journey-qa/app.yaml.
Useful intents:
validate_corpusmatrix seed=... matrix_personas=primary|alldriver_replay_smoke scenario=... persona=... seed=...driver_replay_sweep persona=... seed=...start project=... persona=... seed=...load run_dir=...handoffattachrecordblockerdriver_eventvalidate_matrix_strictreviewvalidate
Prefer the story as the write surface when an operator session is attached.
Use CLI fallback when the story session is unavailable.
Improvement Loop
When refining the pipeline:
- Identify the missing proof from
review.json, validation output,
driver-handoff.md, or a failed flow.
- Patch the smallest durable surface: catalog, runner, story, driver agent, or
this skill.
- Add or update a deterministic flow/cassette/replay check.
- Re-run
--validate-corpus, --driver-replay-sweep, and product-journey
story flows.
- Commit only the product-journey slice, leaving unrelated workspace dirt
untouched.
