Orchestrator checklist to verify worker outputs against the approved plan before accepting artifacts. Phase 2 of orchestrator/worker context enforcement (#2020).
Orchestrator checklist to verify worker outputs against the approved plan before accepting artifacts. Phase 2 of orchestrator/worker context enforcement (#2020).
When a worker (subagent, Codex, Gemini, or parallel terminal) returns results,
the orchestrator MUST verify those results against the approved plan before
accepting them. This skill provides the verification protocol.
Why This Exists
Workers optimize for task completion, not plan adherence. Without verification:
Workers skip optional deliverables that the plan required
Workers modify files outside their assigned scope (contention risk)
Workers produce code that passes tests but drifts from the spec
Orchestrators rubber-stamp results because the output "looks right"
When to Use
After any worker/subagent returns results
After overnight batch runs (morning review)
Before merging worker-produced branches
Before closing a GitHub issue completed by a worker
Verification Checklist
For each worker output, the orchestrator checks:
1. Scope Alignment
Files changed match the plan's "Files to Change" list
No files outside the worker's assigned scope were modified
No files from other workers' negative write boundaries were touched
2. Acceptance Criteria
Each acceptance criterion from the approved plan is satisfied
Evidence exists for each criterion (test output, file content, command result)
No acceptance criteria were silently dropped
3. Test Coverage
Tests were written/updated as specified in the plan
Tests pass (verify via uv run pytest or equivalent)
Test names and coverage match the plan's TDD test list
4. Artifact Completeness
All deliverables listed in the plan are present
Documentation updates (if required by plan) are included
Commit messages follow conventional format and reference the issue
4A. Generated Artifact Freshness
For generated artifacts (manifests, reports, schemas, indexes, lockfiles, static outputs):
Prefer regenerating artifacts from the currently checked-out generator/validator over restoring stale artifact snapshots from patches or archives.
If recovering from an archived patch, treat restored generated outputs as a bootstrap only: rerun the committed generator before validation or commit.
Verify the generator CLI and validator CLI from the current checkout before rerunning old commands; stale renamed flags can create false failures.
When validation reports mass schema-version or missing-field failures across generated outputs, assume generator/artifact schema drift first, regenerate artifacts, then re-run targeted validation before editing validator expectations.
Do not close an issue on recovered generated artifacts until the artifacts have been regenerated or proven byte-for-byte compatible with the current generator and validator contract.
After pulling/rebasing while generated reports or schemas are in flight, re-run targeted tests that reference canonical generated paths; patch tests to discover the latest canonical artifact or regenerate test fixtures instead of hardcoding stale date/version filenames.
Treat a passing validator CLI plus a failing artifact test as a fixture-contract drift signal until proven otherwise: inspect whether the test points at an older generated report/schema path before changing validator semantics.
When the user asks to locate an existing report/output artifact from prior work:
Search prior session records for the exact artifact path and issue context if the local checkout does not contain the sibling repo or output tree.
Verify the artifact against the remote canonical repo using GitHub Contents/API metadata (path, size, sha, download_url) before telling the user it exists.
Keep the distinction explicit: "remote artifact exists" is not the same as "local checkout exists". If the repo is absent locally, say that follow-up edits require clone/sync first.
For generated HTML reports, capture the report path, raw URL, owning issue, size, and SHA; this is enough for a fresh follow-up prompt to resume without regenerating the artifact.
Do not retry the same missing local path repeatedly. After one missing-path failure, inventory available repo roots and switch to session/remote verification.
When a worker claims to have moved, removed, or preserved a nested Git checkout under a parent repo:
Verify source and destination roles independently: remotes, branch, HEAD SHA, upstream ahead/behind, and dirty status.
If deleting a duplicate checkout, require matching canonical remote and HEAD SHA, clean status, and 0 0 ahead/behind unless the plan explicitly preserves divergent work.
If preserving untracked/dirty artifacts from a nested checkout, copy only intended relative paths and verify byte-identical copies (cmp/checksum) before deletion.
Before rm -rf or mv, scan /proc/*/cwd for processes under the source path; identify and clear stale cwd holders, then repeat the scan.
After the move/delete, re-run destination clean/upstream checks and parent-repo path status; do not accept worker self-report alone.
For delegated default-branch pushes, expect some harnesses to require explicit scoped authorization; the orchestrator may push directly only after verifying the exact commit and files.
4D. Machine-Readable Evidence Sidecars and Redaction Safety
When a readiness script, dashboard, or generated report accepts machine-readable evidence sidecars from local or remote hosts:
Validate evidence shape fail-closed before trusting nested data: required top-level keys, checks object type, expected check keys, and required per-check fields such as status, message, and details.
Treat missing evidence and malformed evidence as distinct blockers; both should block remote readiness, but use separate codes/messages so tests and operators can diagnose the cause.
Add tests for both missing evidence and malformed/partial evidence; do not stop at the happy path where the sidecar exists.
Run output through the production redaction path in tests when blocker codes are part of the emitted artifact; redactors can mutate seemingly harmless machine codes.
Keep machine-readable blocker/status codes redaction-safe: avoid substrings like token, secret, api_key, password, credential, and ambiguous words that may match broad secret scanners. Prefer neutral codes like repo_placement_missing and repo_placement_malformed over repo_placement_evidence_*.
Do not weaken secret redaction just to preserve a diagnostic code. Rename the code or split human-readable detail from machine-readable status.
