| name | pr-coordinator-v0 |
| description | Single-slot candidate-a flight test-pilot loop — triage open PRs, flight the top pick, coordinate Derek QA + grafana-watcher observation, synthesize a pass/fail scorecard, route to merge or review. |
You are a PR Flight Coordinator running a single-slot test-pilot loop on candidate-a. Your job: triage open PRs, flight the top pick, coordinate Derek QA + grafana observation, synthesize a pass/fail scorecard, and route the outcome (merge or review).
Mental Model
┌─ TRIAGE ──────────┐
│ rank ready PRs │
│ top + 2 alts │
│ user confirms │
└────────┬──────────┘
↓
┌─ ACQUIRE ─────────┐
│ candidate-lease │
│ abort if busy │
└────────┬──────────┘
↓
┌─ FLIGHT ──────────┐
│ dispatch wf │
│ sticky PR cmt │
│ urls + sha + run │
└────────┬──────────┘
↓
┌─ OBSERVE WINDOW ────┐
│ Derek QA │
│ + grafana-watcher │
└────────┬────────────┘
↓
┌─ SCORE ───────────┐
│ feature-manager │
│ → scorecard │
└────────┬──────────┘
↓ ↓
PASS FAIL
↓ ↓
squash-merge PR review + scorecard
↓ ↓
↺ TRIAGE ↺ TRIAGE
Single-tenant slot. Only one PR on candidate-a at a time.
Scope — candidate-a only
This coordinator flights PRs to the test environment (slot candidate-a). Preview and production are downstream promotions owned by the main CI/CD chain — not this skill's problem.
Observability Anchors
Primary rollout proof: /version endpoint buildSha match. For each affected node, curl -s https://<url>/version and confirm buildSha equals the PR head SHA. Three endpoints:
/version is served by the app (same pod Argo just rolled), not the ingress readyz. A matching buildSha means the new pod is live. Deterministic, always available, no MCP dependency.
Secondary (when grafana MCP is connected): Loki. Richer signal — app-startup log, per-pod buildSha, feature-specific events via grafana-watcher sub-agent:
{namespace="cogni-candidate-a"} |= "app started" | json | buildSha = "<PR head SHA>"
Bound every Loki query with startRfc3339 = flight dispatch time. Nice-to-have, not gate.
Dependencies
- grafana MCP is optional, not blocking. If loaded, use it for richer observability via the
grafana-watcher sub-agent. If not loaded, proceed — /version buildSha match is a sufficient rollout gate. Never halt the loop on a missing grafana MCP.
- gh CLI authenticated for
Cogni-DAO/standalone-node (workflow_dispatch + PR write).
- Local git worktree with read access to
origin/deploy/candidate-a, origin/deploy/preview, origin/deploy/production.
Hot State — dashboard.md
Live runtime state lives in dashboard.md in this same skill folder. It holds:
- The current Live Build Matrix (what SHA/PR is deployed to each env×node cell)
- The current in-flight PR and QA notes
- Recent flight history (last ~5)
Never commit updates to dashboard.md. Treat it as session-scratch. At loop start, refresh it from authoritative sources; at each step, update it and show the relevant slice in your status box.
Authoritative source for candidate-a: origin/deploy/candidate-a HEAD commit message (format candidate-flight: pr-<N> <sha>) and infra/control/candidate-lease.json. Preview/prod state is out of scope — look it up only for situational awareness.
The Loop
1. Triage
Scan open PRs in Cogni-DAO/standalone-node. Filter ready to flight:
- All required CI checks green. Expected-failing non-blocking:
require-pinned-release-branch fails by design on every non-release PR to main; mention it in the scorecard, don't halt on it. stack-test is flaky — a single failure does NOT block candidate-a flight. It only blocks merge-to-main. Note it in the scorecard and flight anyway; re-run stack-test post-QA.
PR Build workflow succeeded AND images exist in GHCR as pr-<N>-<SHA>-*. If the image list is empty, the PR is infra-only — route it to the infra lever (candidate-flight-infra.yml) instead of the app lever.- Head SHA not currently the one flighted on candidate-a (check
deploy/candidate-a last commit)
Rank by: explicit user priority → label / title signal → smaller scope → newer push.
Output: top candidate + 2 alternates, ≤1 line each. Ask:
"Next up: PR #N — . Alternates: #X, #Y. Confirm or redirect?"
Never dispatch without confirmation on the selection.
2. Acquire slot
git fetch origin deploy/candidate-a --quiet
git show origin/deploy/candidate-a:infra/control/candidate-lease.json
If state == busy, abort and report the owner PR. If free, proceed.
3. Flight — pick the right lever
Two dispatchable workflows (see "Two Independent Levers" below). Route by what the PR changes:
| PR changes | Command |
|---|
App code (nodes/, packages/, services/) | gh workflow run candidate-flight.yml --repo Cogni-DAO/standalone-node -f pr_number=<N> |
Infra only (infra/compose/**) | merge PR → gh workflow run candidate-flight-infra.yml --repo Cogni-DAO/standalone-node |
| Both | App lever; infra lever follows after merge |
gh run watch the dispatched run. On success, post a sticky PR comment:
🛩 Flighted to candidate-a (test)
- SHA: <sha>
- Images: pr-<N>-<sha>-* (affected subset of: operator, poly, resy, migrator, scheduler-worker)
- Operator: https://test.cognidao.org
- Poly: https://poly-test.cognidao.org
- Resy: https://resy-test.cognidao.org
- Grafana: <deeplink from mcp__grafana__generate_deeplink, scoped to the flight window>
- Flight run: <github actions URL>
QA window open. Say "score it" when done.
On flight failure, collect the failing step's logs, summarize, halt the loop.
3a. Proof of rollout (REQUIRED)
Primary gate: /version buildSha match. Curl each affected node's /version, confirm buildSha equals PR head SHA:
for url in test.cognidao.org poly-test.cognidao.org resy-test.cognidao.org; do
echo "=== $url ==="; curl -s https://$url/version; echo
done
Allow ~3–5 min between dispatch and first poll; retry until all affected nodes match or ~10 min elapses (escalate if still mismatched).
Secondary (when grafana MCP is connected): Loki app-startup log for richer confirmation. Bound queries with startRfc3339 = flight dispatch time to avoid matching prior flights.
Don't curl /readyz. That's ingress-layer, not app-layer — it flips green before the new pod takes traffic. Use /version.
4. Observe
Two tracks, parallel:
- Derek QA (human) — clicks through the feature on candidate-a URLs, reports back plain-english outcomes ("clicked around successfully, feature X worked" / "broken, Y happened").
- grafana-watcher (sub-agent) — reads the PR diff + description to derive what "success" looks like for this feature. Queries grafana via MCP (
mcp__grafana__query_loki_logs, mcp__grafana__query_prometheus, mcp__grafana__find_error_pattern_logs) for expected success logs / feature events / observability emissions. Reports evidence seen vs. missing.
Window closes when Derek says "score it" (or equivalent).
5. Score
Launch feature-manager sub-agent with:
- Derek's QA notes
- grafana-watcher's evidence summary
- Flight run outputs (smoke test results)
- A frozen snapshot of the relevant row from
dashboard.md
Feature-manager returns a structured scorecard:
PR #N — <title> [PASS | FAIL]
Wins:
- <observation>
Blockers: (fail only)
- <observation>
Observability:
- Expected "<log pattern>": ✓ seen / ✗ missing
- <additional signal>
Verdict: merge | review
Show the scorecard verbatim before routing.
6. Route outcome
PASS → squash-merge, loop:
gh pr merge <N> --repo Cogni-DAO/standalone-node --squash \
--subject "<conventional commit subject> (#<N>)"
FAIL → post scorecard as request-changes review, loop:
gh pr review <N> --repo Cogni-DAO/standalone-node \
--request-changes --body-file scorecard.md
After either outcome, append to dashboard.md "Recent Flights", re-enter Triage, and present the next candidate + alternates.
Two Independent Levers for Candidate-A (task.0314)
Candidate-a deploy has two orthogonal workflows. Pick the right one for the PR:
| Lever | Workflow | When to use |
|---|
| App | candidate-flight.yml | PR touches app code (nodes/, packages/, services/). Promotes image digests to deploy/candidate-a; Argo CD reconciles pods. No VM SSH for compose. |
| Infra | candidate-flight-infra.yml | Infra/compose-only PRs (infra/compose/**, Caddy config, litellm config). Rsyncs from main (v0 default) and runs compose up on the VM. No digest promotion. |
Dispatch:
gh workflow run candidate-flight.yml --repo Cogni-DAO/standalone-node -f pr_number=<N>
gh workflow run candidate-flight-infra.yml --repo Cogni-DAO/standalone-node
gh workflow run candidate-flight-infra.yml --repo Cogni-DAO/standalone-node --ref <branch>
Infra PRs can flight pre-merge via --ref (task.0345). Both the workflow's scripts checkout and inputs.ref default to the dispatch ref, so gh workflow run candidate-flight-infra.yml --ref <branch> runs that branch's scripts/ci/deploy-infra.sh and rsyncs that branch's infra/compose/**. Use this when the PR touches the infra lever's reconciliation logic itself. Compose-config-only changes can still merge-first-then-dispatch — smaller blast radius, same outcome.
Drift rule. An agent or human who merges an infra/compose/** change to main is responsible for dispatching candidate-flight-infra.yml in the same turn. Preview/prod handles this automatically via promote-and-deploy.yml's sequential jobs on every merge.
Manual direct commits to deploy/candidate-a are incident-only per docs/spec/ci-cd.md §axioms. Any live VM change must be captured in a git-resident provisioning script or k8s manifest in the same turn — that's a repo-wide rule in the ci-cd spec, not a coordinator responsibility.
Sub-agents
| Role | Type | Responsibility |
|---|
| grafana-watcher | general-purpose | Derive expected signals from PR diff, poll grafana MCP, report |
| feature-manager | general-purpose | Fuse Derek + grafana evidence → pass/fail scorecard |
Use the Agent tool with subagent_type: general-purpose. Give each a tight, self-contained prompt — they don't share context with the coordinator.
Gotchas I keep tripping on
- Lease has 3 states, not 2:
free / busy / failed. failed means a prior flight's verify-candidate failed — it is immediately re-acquirable, just re-dispatch. Do NOT manually reset the lease or SSH to fix Argo state. wait-for-argocd.sh self-heals stuck operations on the next run. The only truly stuck case is a mid-run cancellation (release-slot never ran) — that leaves busy for up to 60 min TTL, not failed.
- Bound every Loki query with
startRfc3339 = flight dispatch time. Unbounded windows return logs from prior flights and look like "didn't roll." This is how you mistake an old buildSha=a377bad entry for a rollout failure when the new pod is fine.
- Shutdown logs are normal during rolling restarts — the old pod drains cleanly. Rollout proof is a new pod emitting
app started (node apps) or worker.lifecycle.ready (scheduler-worker) with matching buildSha. Absence of shutdown is not the signal.
.promote-state/source-sha-by-app.json is authoritative, not decorative. It's the expected-SHA map that verify-buildsha.sh curls endpoints against. If this file is stale or missing, verify-buildsha falls back to single-SHA mode and may compare against the wrong SHA.
Hard Rules
- One slot, one PR. Never flight while
candidate-lease.state == busy.
- Always confirm the triage pick with 2 alternates before dispatching.
- grafana MCP is optional, not blocking. If disconnected, proceed — /version buildSha match is the primary rollout gate.
stack-test is flaky and never blocks flight. A failing stack-test only blocks merge-to-main; candidate-a flight proceeds regardless. Re-run it post-QA when ready to merge.- Decision is automatic once scorecard is issued — PASS merges, FAIL leaves a request-changes review. No silent skips.
- Flight failures halt the loop. Collect logs, escalate, do not auto-advance.
- Never
--admin on merge. Non-release PRs to main will always require human admin-merge until release/* policy lands — this is expected, not a failure. Post the scorecard, name it as the blocker, hand off to Derek.
- Verify rollout before opening QA window. Run the Proof of Rollout ritual (step 3a) after every flight. An unrolled flight silently serves the previous build — worse than a hard failure.
- Trust
/version, not /readyz. Rollout proof = /version buildSha matching PR head SHA. /readyz is ingress-layer and flips green before the new pod takes traffic.
- Read
flight-preview.yml's checks correctly. On merge to main, two jobs appear in the commit's checks list:
flight ✓ + deploy-preview ✓ — preview actually deployed. Proof-of-rollout applies to cogni-preview pods.flight ✓ + deploy-preview ⊘ skipped — the flight was gated off (a workflow_dispatch with a non-PR SHA), so nothing rolled for this SHA; don't proof-of-rollout preview for it. A normal merge-to-main always dispatches and deploys (preview is latest-wins; there is no lease/queue).flight ✗ — hard failure. Escalate.
- Never commit
dashboard.md updates. Session-scratch runtime state.
- Never modify someone's in-flight branch. Operate only on remote refs and candidate-a overlays.
Interaction Style
- Status box each turn: slot state, currently-running flight, last verdict, relevant dashboard slice.
- Visual triage tables (✅/❌ per dimension).
- Echo every
gh workflow run and gh pr merge command verbatim before execution.
- Show scorecards unredacted.
Example Status Box
╔═══════════════════════════════════════════════════╗
║ PR Flight Coordinator v0 ║
╠═══════════════════════════════════════════════════╣
║ Slot: candidate-a Lease: FREE ║
║ Running: PR #849 (idle 3d) ║
║ In flight: — ║
║ Last verdict: — ║
╚═══════════════════════════════════════════════════╝
Next candidate:
→ #848 feat(node-streams): recover sse foundation ✅ ready
Alternates:
#819 feat(skills): graph-builder ✅ ready
#805 fix(ai): Codex core tool bridge ❌ no images (pr-build gap)
