| name | task-report |
| description | Simulate, verify, and report ONE scenario from the refined E2E test suite (artifacts/scheduler/architect/04-testing-scenarios-refined) — run it on a live Lima host, capture the per-tenant timeline store + serial/journald/DB evidence, build a Schema-1 timeline.json (events[]/steps[]/checks[]), validate it, and derive report.md from task-report-template.md. Use when asked to run/verify/report a scheduler scenario that authors a task and runs a job (a `🅣 task` scenario). For control-plane / tier-gate scenarios that author NO task and boot NO VM, use policy-report instead. The timeline document is the source of truth; the report is a projection. |
Where things live (repo-root paths — this skill runs from the repo root):
- The shared reference:
_timeline/HOWTO.md
— environment (§1), the standard opening + canonical fleet (§2), the six assertion rules + capture
traps (§3), config knobs (§5), the store + event catalog + dead ids (§6), reporting discipline (§8).
This skill does not restate it — a fact found in both places is a bug (the one-home rule); when
they disagree, the HOWTO and ultimately the repo root win.
- The task path (lifecycle,
--dir, cancel semantics, serial logs, cleanup) lives in this file,
below — it has exactly one consumer, so it lives with the procedure.
- The output contracts:
task-report-template.md (report structure) ·
telemetry-timeline.schema.json (Schema 1 — timeline.json).
(timeline-render.schema.json there is Schema 2, a future-UI render contract — read for lane
vocabulary, never produce documents against it.)
- Upstream (authors + bundles the task this skill runs):
_prompts/author-and-bundle-tasks.md.
- The batch driver:
_prompts/run-report-scenarios.md — invokes this skill per scenario, owns halt-on-failure.
- Scenarios, profiles, outputs: the suite
artifacts/scheduler/architect/04-testing-scenarios-refined/ — scenario files by number, _profiles/, and per-scenario outputs under _timeline/_reports/<report-run>/<NN-slug>/ (the dynamic <report-run> is resolved & confirmed in Step 2 — never hard-coded).
Task-scenario simulation & reporting — the task-report skill
What this skill is for — and what it is NOT
This skill handles the 🅣 task scenarios: a task — authored + bundled upstream by the authoring
prompt — is deployed, a job runs inside a VM, and the proof is the run's telemetry + serial + DB
terminal status.
This skill starts at deploy. It never authors, scaffolds, or edits task code, and never runs
task:create — that is the upstream prompt's job. If the task is missing, or a run fails because the
task is wrong, the fix goes back there (Step 6), not here.
For the 🅟 policy — no task control-plane / tier-gate scenarios (no task authored, no VM booted,
the proof is an HTTP admission verdict), use the sibling policy-report skill instead. The README's
Kind column is the authority on which scenario is which — check it, don't guess.
Outputs — in _timeline/_reports/<report-run>/<NN-slug>/
| File | Contract | Audience |
|---|
timeline.json | Schema 1 — scheduler.telemetry-timeline/1 | machines — the single source of truth |
report.md | task-report-template.md | humans — derived from the timeline document; verdict first, tables newest-first |
attachments/* | registered in the document's attachments[] | evidence: the store snapshot (REQUIRED), _control slice, journald capture, serial logs, DB dumps |
The timeline document is the truth; the report renders it. Every number, verdict, and claim in the
report traces to an event seq, a checks[] entry, or a document field. If the report says something
the timeline does not, the report is wrong by definition. (The general discipline — evidence-not-memory,
attachment registration, gaps, immutability, FAIL classification, no secrets: HOWTO §8.)
The task path — deploy, run, inspect (this skill's own reference)
Who owns which verb. create → validate → bundle is authoring — owned by the
authoring prompt
(it writes config.ts/index.ts and gets the bundle green, no live host needed). deploy → run → capture → report is this skill, which assumes the task is already authored + bundled. Task code
is hand-written only inside the authoring prompt, never here.
<slug> is the scenario's hard-coded Slug (its filename stem); <task-version> is the batch
stamp under tasks/testing-scenarios/ (resolved in Step 2 — never hard-coded). Same verbs every
time — only config.ts and the run count differ:
./scheduler task:deploy <slug> --dir ../../tasks/testing-scenarios/<task-version>
./scheduler task:run <slug> --dir ../../tasks/testing-scenarios/<task-version>
Always pass --dir ../../tasks/testing-scenarios/<task-version>. Every ./scheduler invocation
runs with its cwd forced to the cli workspace dir (execFileSync(..., { cwd: workspace }) in
cli/bin/scheduler.mjs), so --dir is always resolved from apps/cli/ — never from wherever
you typed the command. The bare default (cli/.env's SCHEDULER_TASKS_DIR=../../tasks) lands one level
too shallow, at tasks/<slug>/; a stray --dir ../tasks lands one level too deep, at
the repo root, tasks/<slug>/. --dir ../../tasks/testing-scenarios/<task-version> is the one that's
actually correct.
Inspect (run from the repo root; note the jq path below is one ../ shallower than the
--dir flag — it's read by your own shell, not resolved by the CLI shim):
DEP=$(jq -r .deployment_id ../tasks/testing-scenarios/<task-version>/<slug>/.scheduler/state.json)
./scheduler run:list --deployment-id "$DEP"
./scheduler run:get <run-resource-id> --full
./scheduler run:cancel <run-resource-id>
./scheduler run:cancel <run-resource-id> --force
./scheduler run:cancel <run-resource-id> --grace 30000
task:run targets the latest deployment by default; pass --deployment <id> to pin one. Note the
flag asymmetry: task:run --deployment <id> vs run:list --deployment-id <id>. There is no
run:create / deployment:create — runs come from task:run, deployments from task:deploy.
Multiple runs. task:run submits one runs row from config.run.payload. To exercise a queue /
warm reuse, fire it N times. The run count in a scenario is binding — see Step 3.
⚠️ A bidirectional task runs ONE VM at a time (ADR-08d D1, 2026-08-10)
Submitting a second run while one is live does not boot a second VM: by default it queues behind
the live one (--on-busy=queue, the default) or fails fast with --on-busy=reject. network.type
none and egress are unaffected. Two consequences:
- Any assertion needing two concurrent VMs of one bidirectional deployment is currently unprovable.
Do not write it, and do not "fix" a scenario that has one by dropping the assertion — mark it
⛔ BLOCKED-BY-D2 (scenarios 66 and 30's G1 variant did) so the coverage is deferred, not lost.
- A queued run looks like a hang. It sits
PENDING until the live VM finishes rather than erroring.
Behind a PINNED VM it is refused outright instead — a pinned VM has no idle timeout, so queueing
could wait forever.
The limit is temporary in intent but not scheduled for removal (ADR-08d Q5) — treat it as the
standing shape of the suite.
⚠️ run:cancel is GRACEFUL BY DEFAULT (ADR-13, 2026-08-12) — this changes what you assert
A duration:'long' run does not stop when you cancel it. The guest is asked to wind down on its next
heartbeat ACK and the run stays RUNNING until it flushes or the grace window (cancel_grace_ms,
default 15 s) expires. Asserting a terminal status immediately after cancelling a long run is a false
red — you are reading it mid-drain.
| You cancelled… | Expect |
|---|
a long run, no flags | 200 {status:'RUNNING', draining:true, grace_ms:<armed>} → then COMPLETED if the guest flushed in time, else CANCELLED at the deadline |
| a short/one-shot run | immediate CANCELLED — no carrier exists, so no window is armed |
anything with --force | immediate CANCELLED |
- A deadline-ended drain settles
CANCELLED, never FAILED, and is never retried — assert
failure_attempt_count = 0 and the absence of run.failed, not just the status.
- A pinned VM no longer requires
--force for a graceful cancel (OQ-3): the drain leaves the VM and
the pin alive. --force still 409s without it, because that path destroys the VM.
- The drain needs a guest that announced
cancel-ack (OP_HELLO). On an un-rebuilt rootfs vm:list
shows guest: null and every cancel is an immediate kill — if a drain scenario "does nothing", check
that first, then re-run 03-build-base-rootfs.sh.
Serial VM logs — the guest's own words (post-mortem)
The guest worker emits only log(), onto the VM serial console, captured to
/var/log/scheduler/vms/<correlationId>.log. The orchestrator's tailer ingests these, re-stamps
src.trust='guest', pins identity from the authoritative runs row, and re-emits them (so guest logs
also appear in the store, §6A). A worker-origin event/metric is treated as forged and dropped —
trust is host-stamped.
cat /var/log/scheduler/vms/<CORR>.log
Teardown never deletes these (vm-cleanup.ts preserves them), but a reaper bounds the directory: age
30 d + total 4 GiB, oldest-first, keeps newest (per-file cap 16 MiB). Copy the one(s) you need into
attachments/ promptly; mark raw serial excerpts visibility: "operator".
Warm reuse, from the store + DB (the corrected proof)
cat "$STORE" /var/log/scheduler/timeline/_control.jsonl \
| jq -r 'select(.payload.attrs.name=="vm.provisioning") | .ctx.vm_id' | sort -u | wc -l
jq -r 'select(.payload.attrs.name=="run.claimed") | .payload.attrs.reuse' "$STORE" | sort | uniq -c
sudo -u postgres psql -d scheduler -tAc "SELECT vm_id, COUNT(*) FROM runs WHERE deployment_id='$DEP' GROUP BY vm_id;"
Warm reuse ⇒ one vm.provisioning (one vm_id) serves N runs, the later ones showing run.claimed
reuse=warm; cold-only ⇒ one vm.provisioning per run and no run.claimed. (vm.booted is DEAD —
HOWTO §6A.)
On-host footprint — the cleanup checklist (report §10)
| Resource | Path | After teardown |
|---|
| code images | /var/lib/scheduler/images/<tenant>/<slug>/<contentHash>/code.ext4 | deleted (per-VM), image tree persists |
| task artifacts | /var/lib/scheduler/artifacts/<jobId>.artifact (flat, not tenant/slug) | present (result store) |
| Firecracker API sockets | /var/run/scheduler/vms/<correlationId>.socket (keyed by correlationId) | deleted |
| vsock control UDS | /var/run/scheduler/ctl/<vmId>.sock (keyed by vmId; + .sock_<port> listener) | deleted |
| ephemeral drives | /var/lib/scheduler/vm-tmp/{scratch-,code-}<vmId>.ext4 | deleted |
| ingress xDS/TLS | /var/lib/scheduler/ingress/{lds.json,cds.json,tls/*} | route removed (ingress.route-removed) |
| egress xDS | /var/lib/scheduler/egress/lds.json | policy re-published (hot reload) |
| serial log | /var/log/scheduler/vms/<correlationId>.log | present (reaper: 30 d / 4 GiB, per-file 16 MiB) |
(The shared surfaces — timeline store, metric log, DB — and the GONE paths: HOWTO §7.)
Document-authoring rules (this skill's own discipline)
These are about building a correct Schema-1 document. How the scheduler behaves is the HOWTO's job;
the general reporting discipline is HOWTO §8 — neither is restated here.
- Fill the timeline first. Build
events[] (ascending by (occurred_at, seq)) before any prose —
it surfaces contradictions early. seq is per-document, dense from 1, never renumbered.
- Two layers, never mixed.
events[] are mechanical facts ({seq, ts, type} + flat scalar data);
steps[] is a narrative overlay that references facts only via covers ({events:[seq…]} or
{range:[from,to]}). Prose lives in step names/notes and document notes, never in event data.
- Every assertion is one
checks[] entry — expected vs observed, with class (derivable = machine
can recompute · scenario-intent = what the scenario promised · human-verdict = you judged),
verdict_by, and anchors (events:[seq…] / steps:[id…]).
- Every run's terminal status is a check with a rationale. One
checks[] (class scenario-intent)
per run asserting its expected terminal status (COMPLETED/FAILED/CANCELLED), and
subject.runs[].purpose states why (e.g. "targets a 30 s cap and sleeps 45 s — reap is the point").
On mismatch, report §6 renders an investigation-clue block (first divergent seq + likely cause) — never
silently reconcile.
events[] are built FROM the store snapshot (the REQUIRED attachment), so document and evidence
cannot diverge. Event type = the payload.attrs.name value; the catalog + dead-id list are
HOWTO §6A — never assert a dead id, never invent a bare system id, never emit sim.*.
- ⚠️ DEDUPE THE SLICE BY
id BEFORE BUILDING events[]. Restarts re-publish guest records (F30 —
the measured numbers and recipes: HOWTO §6A), so a slice spanning a restart turns one fact into two
events[] entries with two seq values — a fabricated event, and every checks[] anchored on the
duplicate cites a fact that happened once. . Keep
the RAW slice as the registered attachment (it is the evidence); dedupe only when deriving ,
and say so in when a restart was involved. And count records, not string occurrences (each
record carries its name twice — HOWTO §6A).
Every host footprint is an EVENT, not just prose
Anything you do to the shared host to make a scenario reproducible is an operator footprint on shared
state — exactly like a run or a VM boot — and it MUST leave a trace in timeline.json, not only a sentence
in report.md. An event first, prose second. Report prose describing a change with no matching event
is a document/report divergence. The x-* namespace is open, so synthesize these:
(a) Config edits — x-config.changed, PAIRED. Any config.yaml/config.ts edit before, during, or
between runs (e.g. scheduler.max_concurrent_vms: 1 for a deterministic warm test):
- Before the edit — emit
x-config.changed (inferred:false, data.actor:"operator", data.file,
data.key, data.before, data.after, data.reason, data.restart:true|false) at the seq where
the edit actually happened.
- After the revert — emit a second
x-config.changed, before/after swapped.
- One
checks[] entry (class derivable) asserting both exist, referencing their seq.
report.md §1 renders it as a small before/after table (seq · field · before · after · reason ·
reverted?) — not a paragraph. §5d holds the fuller table; §1 is the at-a-glance version.
- §4 (timeline digest) and §5d cite the real
seq — never —.
(b) Orchestrator kill/restart — x-orchestrator.killed + x-orchestrator.restarted, PAIRED. Many
async scenarios (crash-requeue, deploy-build-survives-restart, deadletter-redrive) prove their point by
deliberately killing/restarting the orchestrator mid-flight. That is the mechanism under test —
evidence it:
x-orchestrator.killed at the seq you signalled — data.signal (SIGKILL/SIGTERM),
data.pid, data.reason, data.expected_recovery, inferred:false, data.actor:"operator".
⚠️ The signal value must be exact: SIGKILL is a crash (no drain), SIGTERM a graceful
stop (drains) — load-bearing for crash-vs-planned scenarios. And verify you signalled the process
you meant to — the tsx MainPID trap (HOWTO §3) makes a "crash" read as a graceful shutdown.
x-orchestrator.restarted at the seq it came back — data.method (systemctl start/restart), data.down_ms, data.recovered (what the startup sweep reclaimed — e.g.
runs.attempt_count bumped, an outbox row returned CLAIMED→PENDING).
- One
checks[] entry (class derivable) asserting both exist and that the post-restart effect the
scenario promises anchors to their seq.
report.md renders them in the §4 digest and the §5d footprint table, alongside x-config.changed.
The same rule covers every footprint: a tier grant, a manual DB poke, an ingress route change —
event first, prose second. (Ingress changes already emit their own ingress.route-* events; don't
reconstruct those — HOWTO §6A.)
Step 1 — Read the references & confirm preconditions
Read HOWTO.md
(§1 host/API/CLI setup, §2 the standard opening, §3 the assertion rules, §5 config knobs,
§6 the evidence surfaces + event catalog) and this skill's task path section above. Confirm
before you start: orchestrator up (journalctl -u sched-orchestrator), a tenant + API_KEY (and
ADMIN_KEY to publish tiers) — all inside Lima (limactl shell lima-vm); the DB is PostgreSQL, queried
via sudo -u postgres psql -d scheduler -tAc "SELECT …" (HOWTO §1).
Step 2 — Resolve the task version, then name the report-run folder
There are two distinct stamps — do not conflate them (conflating them is exactly how a folder
got mis-named 2026-07-29-v1-live-pg-2026-08-03):
<task-version> (an INPUT) — the batch tag under tasks/testing-scenarios/<task-version>/<slug>/,
e.g. 2026-07-29-v1. You resolve it (below); you never rename it.
<report-run> (an OUTPUT) — the folder under _timeline/_reports/<report-run>/<NN-slug>/ that
holds THIS run's timeline.json + report.md. You name it (naming rule below).
Discover, then confirm — never guess:
ls -1d tasks/testing-scenarios/*/ 2>/dev/null | sort -r
ls -1d tasks/testing-scenarios/*/<slug>/ 2>/dev/null | sort -r
- Exactly one version holds
<slug>/ → propose it and ask the user to confirm ("Run <slug>
from version <X>?").
- Several → list them newest-first and ask which to run.
- None (task not authored yet) → stop and point the user at the
authoring prompt
to create + bundle the batch first.
Set <task-version> to the confirmed answer. Then derive <report-run> — the output folder — by this
rule, do not reuse the task version verbatim:
- Same day, first run of a freshly-authored batch →
<report-run> = <task-version>. This is the
only case where the two stamps coincide.
- Any re-run — a later day, a different environment (live-pg), or a fixes pass → name it
<run-date>-v<n>[-<env>], where <run-date> is TODAY (the day you run it, the leading sort key).
So a live-PG re-run on 2026-08-03 of a 2026-07-29 batch is 2026-08-03-v1-live-pg — never
2026-07-29-…-2026-08-03 (leading with the stale task-batch date, or double-dating, is the mistake
this rule exists to prevent).
Confirm the derived <report-run> name with the user before writing any file (state which
<task-version> it exercises, so the trace back to the task batch is recorded in the report body — §5 —
not smuggled into the folder name). Then use <report-run> verbatim for every _reports/… path, and
<task-version> verbatim for every tasks/… and --dir path.
Step 3 — Read the scenario file
- Confirm the Kind in the README fast-lookup — it should be
🅣 task. If it is 🅟 policy — no task, stop and use policy-report.
- Slug — the scenario's hard-coded
**Slug:** (its filename stem). It names the authored task
directory and is subject.task.slug, so deployment/runs trace back to the scenario.
- Profile — which of
_profiles/{standard,free,network,pinned,batch}.json it uses → publish it
(check first — HOWTO §2) → binding records the tier under test.
- §1 execution block →
binding.axes (the config.ts execution values).
- Assert-the-effect table + Definition-of-done — each assertion becomes a
checks[] entry.
- Readiness (
✅ / ⚠️) → binding.readiness. A ⚠️ degradation is expected behaviour — record
it, don't fix it.
The run count is binding. Take the maximum any assertion implies (an "N distinct vm_ids"
assertion needs N runs, even if a happy-path line shows one). Running fewer times than an assertion
needs means it is not verified — mark it PENDING in checks[] (never silently pass/drop) and say
so in §1 notes.
Step 4 — Deploy, run, and capture
The task is authored + bundled upstream. Confirm it is present (<task-version> from Step 2,
<slug> from Step 3), then start at deploy (commands + the --dir rule: the task path above):
ls tasks/testing-scenarios/<task-version>/<slug>/config.ts tasks/testing-scenarios/<task-version>/<slug>/index.ts
cat tasks/testing-scenarios/<task-version>/<slug>/.scheduler/state.json 2>/dev/null
./scheduler task:deploy <slug> --dir ../../tasks/testing-scenarios/<task-version>
./scheduler task:run <slug> --dir ../../tasks/testing-scenarios/<task-version>
Re-bundle first (./scheduler task:bundle <slug> --dir …) only if the task tree changed since
the author prompt ran; otherwise the existing bundle is authoritative.
Deterministic warm test: set scheduler.max_concurrent_vms: 1 in config.yaml (HOWTO §5) and
sudo systemctl restart sched-orchestrator before running — an operator config knob (tracked as an
x-config.changed event, above), never a task edit.
Capture as you go (cheaper than reconstructing — surfaces and paths: HOWTO §6/§7 + the task path):
DEP=$(jq -r .deployment_id ../tasks/testing-scenarios/<task-version>/<slug>/.scheduler/state.json)
TEN=$(sudo -u postgres psql -d scheduler -tAc "SELECT tenant_id FROM deployments WHERE id='$DEP';")
mkdir -p _timeline/_reports/<report-run>/<NN-slug>/attachments && cd _timeline/_reports/<report-run>/<NN-slug>
journalctl -u sched-orchestrator --since "10 min ago" > attachments/orchestrator.log
jq -c 'select(.ctx.deployment_id=="'"$DEP"'")' /var/log/scheduler/timeline/$TEN.jsonl \
> attachments/store-snapshot.jsonl
jq -c 'select(.ctx.deployment_id=="'"$DEP"'")' /var/log/scheduler/timeline/_control.jsonl \
> attachments/control-events.jsonl
cp /var/log/scheduler/vms/<CORR>.log attachments/serial-<CORR>.txt 2>/dev/null || true
sudo -u postgres psql -d scheduler -tAc \
"SELECT id,status,vm_id,started_at,correlation_id,substr(result::text,1,80),substr(progress::text,1,40),instructions \
FROM runs WHERE deployment_id='$DEP' ORDER BY created_at DESC;" > attachments/db-runs.txt
⛔ *.log IS GITIGNORED — a registered .log attachment never reaches the repo
.gitignore line 6 is *.log. Anything you capture under that extension is silently dropped at
commit time, so a document that registers it — with a real sha256, which makes it look verified —
claims evidence the repo does not contain. Found 2026-08-14: 14 .log evidence files under
_reports/, zero tracked, and one document registering three of them.
Two rules, and they are about naming what you mean:
- Serial logs are EVIDENCE → land them as
.txt. They are the guest's own words, they are small
(26–43 KB), and they are irreplaceable: the host reaper bounds /var/log/scheduler/vms/ at
30 days / 4 GiB, so "fetch it later" expires.
- The journald capture is SCRATCH → leave it
.log, and do NOT register it. It is 20–30 MB of
mostly noise and the store snapshot is the authoritative surface anyway. If a journald excerpt is
genuinely load-bearing, grep the handful of lines you need into a .txt and register that.
⚠️ Do not "fix" this with a .gitignore negation for the evidence tree — that makes the 29 MB
journald captures committable by default, and the next run commits one without noticing.
Verify before you register — one line, and it closes the whole class:
git check-ignore -v attachments/* || echo "OK — every attachment is committable"
Note the wall-clock of each significant action — these become event ts for actions the store doesn't
cover (the CLI emits no telemetry — HOWTO §6A). Temp files stay under
_timeline/_reports/<report-run>/<NN-slug>/, never /tmp.
Step 5 — Build the timeline, then derive the report
- Build
timeline.json first (facts, ascending). Sources, in order of authority: store snapshot
(payload.attrs.name→type, occurred_at→ts, ctx.*→refs, rest of payload.attrs→data) →
the runs table (terminal status incl. COMPLETED, which has no event; instructions; vm_id) →
serial/journald (guest markers, inferred support). Then add steps[], checks[], overall,
sources[], attachments[] (real sha256 — shasum -a 256 <file>), gaps.
- Validate against Schema 1 before any report prose:
npx ajv-cli validate \
-s artifacts/scheduler/architect/04-testing-scenarios-refined/_timeline/report-templates/telemetry-timeline.schema.json \
-d _timeline/_reports/<report-run>/<NN-slug>/timeline.json
(No scheduler timeline:validate command — it was removed.) Fix errors in the document, never by
bending the schema. No validator available → hand-check the required fields (spec, document_id,
generated_at, generator, window, events) and note it in gaps.
- Derive
report.md from the validated document, following
task-report-template.md
section by section (it is authoritative — each section names the Schema-1 field it renders). Introduce
nothing not in the document; if you need to, the fact belongs in the timeline first.
overall is PASS only if every checks[] passed; FAIL if any failed; PENDING if any verdict
is open. Report §1's verdict is this field, verbatim.
Step 6 — Debug a failure, and where the fix goes
When a check fails or a verdict stays open, the document already carries the lead: report §6 renders
the first divergent seq + likely cause. A document is immutable, so a fix is always a new
document_id (with supersedes), never an edit. Classify the cause:
- Environment / operator-config (host not provisioned, wrong tier granted, a
config.yaml knob, a
flake) → fix it and re-run — a new run-id / timeline. Bounded retries, then leave it PENDING.
- Task bug (bad
index.ts/config.ts — the run does the wrong thing) → stop. Do not edit task
code here. Return to the
authoring prompt,
fix + re-bundle, then resume this skill for that scenario.
- System defect / scenario-spec defect → the batch driver's Step 4 owns the full classification +
investigation.md contract; routing: HOWTO §8.
Never silently pass or drop a failed assertion — leave it FAIL/PENDING with the clue.
The one-home rule, applied here: how the system behaves (rules, gates, store, catalog, knobs,
traps) → HOWTO.md.
The report's structure → the
template.
Batch order + halt-on-failure → the
driver prompt.
This file owns the per-scenario procedure and the task path only. A new trap or correction lands in
the HOWTO — never here.