harness-run

name

harness-run

description

Drive one or more control-room workflows end-to-end and diff the capture against the prior run for the same workflow. Surfaces which (persona, step) cells changed and why, so a cascade regression caused by a recent change becomes a structured finding instead of an after-the-fact "I think something broke." Runs inline in the current chat (no subagent fork). Use when the user says "run the harness", "drive a workflow", "verify the cascade", "check for regressions across personas", "run the journey loop", or after a non-trivial change to rights derivation / state-machine guards / Drizzle queries / persona shells.

Harness run + diff (inline)

Perform this inline in the current chat using Bash, Read, and the existing tooling under apps/control-room/. Do NOT spawn an Agent. The skill is one operator + one set of capture artifacts; forking would lose the per-cell signal.

The point: every captured run becomes a regression baseline. Each new run automatically diffs against the prior, so any change that shifted a derived hash, flipped an expect, broke an agreement, or introduced a console error shows up as a tinted cell in the dashboard AND a structured finding here.

When to invoke

• User says: "run the harness", "drive workflow X", "verify the cascade", "run the journey loop", "check that nothing broke across personas"
• After a change to:
  • packages/domain/src/rights/ — visibility, edges, canPerform
  • packages/domain/src/machines/ — state machine guards
  • packages/data/src/queries/ — any query whose result feeds a persona's derived view
  • packages/services/src/mutations/ — anything that emits audit / cascades across personas
  • apps/web/src/app/<persona>/ — any persona-shell route
• Before declaring a multi-file lift complete (the lift loop ships surfaces; this is the cascade-side proof they compose)

Skip when:

• Only docs / comments / type-only refactors changed
• The change is purely a UI polish on a single non-shell surface (use the snapshot script directly instead)

Setup — pre-flight before driving

Run the pre-flight first; bail with a clear message if any step fails. These are all hard prereqs, not heuristics.

1. Postgres reachable — iep_control_room exists at port 5438. The harness will drop+recreate it from the fixture template each run; that's expected and SAFE (it refuses to drop anything not matching the iep_control_room* whitelist). If the DB doesn't exist yet, tell the user to run pnpm db:reset && pnpm --filter @iep/data db:seed:personas && pnpm --filter control-room fixtures:build.

2. apps/web is not running on :3003 — Next.js 16 holds a directory-level dev lockfile, so the harness can't spawn its own apps/web while the operator's is up. Detect via apps/web/.next/dev/lock. If present, surface the PID from the lockfile and ask the operator to stop it OR use shared mode (IEP_WEB_SKIP_SPAWN=1 with the running server pointed at iep_control_room). Don't kill it without explicit OK.

3. IEP_PERSONA_SECRET set on the orchestrator's env — needs to be ≥32 chars and match what apps/web validates with. The canonical pattern: apps/control-room/.env.local carries IEP_PERSONA_SECRET + IEP_CONTROL_ROOM_DATABASE_URL. If missing, tell the operator to copy .env.local.example and restart pnpm --filter control-room dev.

4. Workflows on disk — the user's argument should match a file in apps/control-room/workflows/. "all" runs every workflow sequentially. List them via ls apps/control-room/workflows/*.json if the user didn't name one.

Drive

For each workflow:

IEP_CONTROL_ROOM_DATABASE_URL="postgres://postgres:postgres@localhost:5438/iep_control_room" \
IEP_PERSONA_SECRET="$(grep -E '^IEP_PERSONA_SECRET=' apps/control-room/.env.local | cut -d= -f2-)" \
pnpm --filter control-room run-workflow <name>

• The CLI prints [run-workflow] <id> run <runId> captured (N steps × M personas) on success. Parse the runId from that line.
• On failure (non-zero exit), capture the tail of stderr and surface it directly — failures inside fillFormValues or engine.expects are real bugs in the workflow JSON or in apps/web, not skill flakes.
• Each run takes ~10–30s. Don't poll; the CLI is synchronous.

Diff

After each run, find the prior run for the same workflow:

ls -t .snapshots/control-room/runs/<workflow-id>/ | head -2

The first entry is the run you just produced; the second is the baseline. If only one entry exists, this is the first run on disk and there's no baseline — note that and skip the diff for this workflow.

The dashboard surfaces the diff at:

http://localhost:4444/runs/<currentRunId>?wf=<workflowId>&vs=<baselineRunId>

For the structured finding list, invoke the diff CLI — a thin shell around orchestrator/run-diff.ts that prints in the contract format below. CLI and dashboard share the same diffCell / mismatchKey so a new signal lands in one place:

pnpm --filter control-room run-diff <workflowId> <currentRunId> <baselineRunId>

The key signals per cell:

• derived — derivedHash flipped (most state changes)
• agreement — agreement.ok flipped or mismatch identities changed
• expects — any per-actor pass/fail or observedMarkers changed
• console — console-error count changed (loudest regression signal)
• lifecycle — cell kind flipped (changed ↔ no-change ↔ absent)

Output format

Report inline, structured per workflow, terse:

workflow-id (run <runId> vs baseline <baselineId>)
  ✗ 03-teacher-logs   teacher    derived, expects
                                  expect: queue.contains observation-logged → FAIL (was PASS)
  ✗ 06-after-snapshot admin      derived, console
                                  console: PostgresError column "ieps.foo" does not exist
  ↻ 14 other cells changed by reason: derived×12 lifecycle×2

For each changed cell where reasons include agreement, expects, or console, include the underlying fact (mismatch kind, expect marker that flipped, console error message). Console messages truncate at 160 chars. Cells whose only reason is derived (deliberate side-effect of the change) get rolled up into the "other cells changed by reason" summary — don't list 40 derived-only cells individually.

End with:

• A one-line dashboard URL (http://localhost:4444/runs/...)
• A single judgment line with deduped loud reasons:
  • no signal flipped — clean run (changed == 0)
  • expected change — derived only (no loud reasons fired)
  • unexpected <reasons> regression — investigate (deduped sorted set of loud reasons, e.g. agreement/console, never agreement/console/agreement/console)

Limits

• Don't recommend fixes inside this skill. Surface the finding; the agent / operator decides.
• Don't author new workflows here. If the user wants more coverage, point them at apps/control-room/workflows/ and the workflow-schema.ts Zod validator.
• Don't compare runs across different workflows. The skill is keyed by workflowId and the diff is meaningless cross-workflow.
• The first run of a brand-new workflow has no baseline. That's not a failure — note it and move on.

Cleanup

You are the main session — don't create scratch files. The runs land at .snapshots/control-room/runs/<workflowId>/<runId>/ automatically and are the expected artifact. End with git status to confirm nothing else got staged.