deploy-watchdog

name

deploy-watchdog

description

Watch a develop→main deploy.yml run to completion; if it fails, diagnose the failure, open a fix PR against develop, wait for that PR's CI to go green, and report back so the user can merge. Use when the user says "watch the pipeline", "I just merged dev to main", "watch the staging pipe", "I merged develop into main", "babysit this deploy", "watch deploy.yml", or any variant asking the agent to monitor a develop→main merge on this project.

Deploy Watchdog

End-to-end monitor for the preview-then-promote pipeline on main (see docs/runbooks/deploy.md). The loop below is the contract the user explicitly asked for.

The Loop

┌─ watch deploy.yml on main (triggered by develop→main merge)
│    wait for all jobs to terminate (~25–30 min typical)
│
├─► if GREEN:
│     report success + hand promotion back to the user. DONE.
│
└─► if RED:
      1. identify which job failed (usually smoke-staging; sometimes
         deploy-backend / deploy-frontend)
      2. pull artifacts (Playwright report + test-results) AND raw
         step logs for the failed job
      3. diagnose root cause from artifacts + logs + any relevant
         code/config files — do NOT guess
      4. write a short plan: root cause, minimal fix, why-not-alternatives,
         test plan. Map to a TodoWrite list.
      5. branch from latest origin/develop: fix/<kebab-case-summary>
      6. implement the fix (smallest diff that solves it)
      7. add a capstone lesson to .cursor/tasks/lessons.md covering:
         - the symptom
         - why it was latent
         - the class of antipattern
      8. commit (Conventional Commits), push, open PR against develop
         with a body that includes: summary, root cause, change,
         why-not-alternatives, test plan, lesson pointer.
      9. wait for PR CI to go green (Playwright E2E + lint at minimum)
     10. tell the user: "PR #N is green — you can merge to develop"
         and STOP there.
     11. When user confirms merge (either explicitly or by mentioning
         a new develop→main merge), re-enter the loop at the top.

How to Execute

Step 1 — Find the run

gh run list --workflow=deploy.yml --branch=main --limit=2 \
  --json databaseId,status,conclusion,displayTitle,createdAt

The top entry (status: in_progress) is the one to watch. Grab its databaseId.

Step 2 — Poll until terminal

The pipeline has ~7 jobs. Typical wall-clock:

Job	Typical duration
backend-ci/* (lint, unit, integration)	5–8 min
Wait for in-flight Terraform Apply	<1 min
deploy-backend (build+push+deploy)	17–20 min
deploy-frontend (Vercel preview)	1–2 min
smoke-staging (Playwright + Lighthouse)	4–6 min

Total: ~28–36 min. Poll with:

gh run view <runId> --json status,conclusion,jobs | python3 -c \
  "import json,sys; d=json.load(sys.stdin); \
   print(f\"overall: {d['status']} {d.get('conclusion') or '-'}\"); \
   [print(f\"  {j['status']:12} {j.get('conclusion') or '-':10} {j['name']}\") \
    for j in d['jobs']]"

Use AwaitShell with increasing backoff (e.g. 3 min → 10 min → 9 min → 4 min) rather than tight polling. Terminal states: completed success (green) or completed failure (red).

Step 3a — If GREEN

Tell the user:

• Which run was watched (with URL)
• Per-job timings (nice-to-have)
• That promote.yml is now safe to dispatch

Then STOP. Do not auto-promote — promotion is always a human decision.

Default promote command (auto-discovers the preview URL from the latest green deploy.yml run on main — no copy-paste required):

gh workflow run promote.yml -f confirm=promote -f run_post_promote_smoke=true

The resolve job in promote.yml echoes the resolved preview URL, source run ID, and commit into the run summary BEFORE promote-backend flips traffic — the operator gets the "here's what you're about to flip" moment from the summary instead of from their clipboard.

Only pass -f preview_url=<URL> explicitly when promoting a preview from a non-main branch, or re-aliasing an older preview that's still un-pruned by Vercel.

Step 3b — If RED

1. Identify the failed job from the job list above.
2. Pull artifacts:

   rm -rf /tmp/smoke-artifacts && mkdir -p /tmp/smoke-artifacts && \
     cd /tmp/smoke-artifacts && \
     gh run download <runId> -R joaothomazlemos/rag-concurso
   

   If no valid artifacts found, the failure was BEFORE Playwright ran (preflight, wait-for-alias, Lighthouse, etc.) — fetch step logs instead:

   gh api repos/joaothomazlemos/rag-concurso/actions/jobs/<jobId> \
     --jq '.steps[] | "\(.number) \(.status) \(.conclusion) \(.name)"'
   gh run view <runId> --log-failed | tail -100
   

3. Read artifacts richly: error-context.md in test-results-prod/<test>/ contains the assertion message AND a DOM page snapshot. That snapshot frequently tells the whole story without ever opening trace.zip.
4. Diagnose before fixing. The mandatory questions:
   • What exactly failed? (which assertion, which step, which URL)
   • Why did it fail? (root cause, not symptom)
   • Why didn't prior runs catch it? (latency / latent / blocked by upstream failure)
5. Plan with TodoWrite. Include: root cause, fix, anti-alternatives, test plan, expected CI behaviour.
6. Branch + fix + lesson + PR.
   • Branch naming: fix/<kebab-case-summary> off origin/develop (ALWAYS sync first: git fetch origin && git checkout develop && git pull --ff-only)
   • Smallest possible diff that solves the root cause
   • Always append a lesson to .cursor/tasks/lessons.md — see the existing entries (dated sections) for style
   • Conventional Commit message with detailed body
   • PR against develop (NOT main — main is only reached via the develop→main auto-merge PR)
7. Wait for PR CI — usually Playwright E2E (~4 min) + lint (~1 min). For workflow-only / docs-only changes, CI may show "no checks" due to path filtering — that's still mergeable, but note it explicitly.
8. Report back: PR URL, CI status, and the instruction "you can merge to develop".

Golden Rules

• Never auto-merge to develop or main. The user explicitly reserves both merges (and always promotion). You open PRs; they merge.
• Never amend the user's merge commit. If CI fails on main, open a forward-fix PR on develop — do not force-push / revert main.
• Never skip the lesson. Every red→green cycle is a lesson in .cursor/tasks/lessons.md. Skipping them means the next loop recommits the same class of mistake.
• Pull the artifact before touching code. The DOM snapshot in error-context.md has ended every diagnosis in this project's history faster than running any local command.
• Minimal diff. Every red→green PR in this project has been ≤20 lines of code change (plus a lesson). If the fix feels bigger, stop and question whether the diagnosis is right.
• Watch PR CI, don't merge it. Report green; let the user merge.

Prior Executions (pattern bank)

Track record of this loop, newest first. When you hit a red, check this table first — the next failure is often adjacent to the last one.

Date	Failure	Job	Root cause	Fix PR
2026-04-22	Lighthouse a11y score 0.84 (<0.85)	smoke-staging	text-gray-400 on white (contrast 2.6); hover:underline-only link	#145 (gray-600 + always underline)
2026-04-22	Lighthouse 404 on /login	smoke-staging	Vercel preview URLs don't SPA-fallback; Lighthouse does direct HTTP GET	#143 (one-line: audit / instead)
2026-04-22	Chat round-trip length floor > 3	smoke-staging	Nova Micro obeyed prompt with "Ok." (3 chars) = placeholder length	#141 (.not.toBe("...") divergence check)
2026-04-22	psycopg3 prepared statement _pg3_3 already exists	smoke-staging	pgbouncer transaction mode + psycopg3 auto-prepare	#138 (prepare_threshold=None via create_pg_pool factory)
2026-04-22	Cloud Run serving backend without auth	smoke-prod (prod)	SUPABASE_URL/SUPABASE_SECRET_KEY missing from Cloud Run env	#135 (Terraform locals.tf)

LHCI Report Extraction

When smoke-staging fails at Lighthouse (step 12 in smoke-prod.yml), no artifacts are uploaded — the report ships to Google's temporary public storage. Find the URL in the step log:

gh run view --job <jobId> --log | grep "Open the report at"

Download and extract scores + failing audits with this Python snippet:

import re, json
with open('/tmp/lh-report.html') as f: content = f.read()
m = re.search(r'window\.__LIGHTHOUSE_JSON__\s*=\s*(\{.*?\});\s*</script>',
              content, re.DOTALL)
data = json.loads(m.group(1))
for key in ['accessibility', 'seo', 'performance', 'best-practices']:
    print(f"{key}: {data['categories'].get(key, {}).get('score')}")
a11y = data['categories'].get('accessibility', {})
for ref in a11y.get('auditRefs', []):
    a = data['audits'].get(ref['id'], {})
    if a.get('score') is not None and a.get('score') < 1 and ref.get('weight',0) > 0:
        print(f"  [weight={ref['weight']}] {ref['id']}: {a.get('title')}")
        for it in a.get('details',{}).get('items',[])[:3]:
            n = it.get('node',{})
            print(f"    selector: {n.get('selector','')[:200]}")

This gives the exact DOM selector, offending colours, and WCAG rule reference in <5 seconds — which is the diagnostic that closed #145 in under 10 minutes end-to-end.

Invariants you can trust

• deploy.yml:134 passes the staging-tagged backend URL to smoke — so smoke-staging exercises the exact not-yet-promoted revision.
• smoke-prod.yml:84 honours the backend_url input (falls back to vars.BACKEND_URL on manual dispatch, which IS prod).
• Playwright artifacts upload on failure() || cancelled() — always present for Playwright-caused failures (step 8).
• Lighthouse step is step 12 and uploads nothing; for Lighthouse failures you want the raw step log, not artifacts.
• Promotion is behind promote.yml with environment: production (reviewer-gated). Nothing in this skill's loop can touch prod traffic.