| name | 15-service-health |
| description | Investigates a deployed service in two layers: platform infra health (API up, DB migrations, secrets, deploy drift, GitHub main CI green via H0ci) and live behavior (API smokes, E2E tiers). AskQuestion-driven depth and budget before running checks. Test-driven on user failures (repro test red → confirm → investigate → fix via 14-hotfix). Opens docs/bug-reports/BUG-*.md for code bugs. Use for production health, ambiguous API/DB errors, periodic ops, post-deploy verification, or confirming main CI passes after merge/hotfix.
|
15 — Service Health
Investigate the live deployment: correct version, database ready, API paths working,
and alignment with docs/deploy.md §Integration — without assuming a hotfix is required.
Code failures: bug-investigation → docs/bug-reports/BUG-*.md
tests/bugs/test_bug_*.py before 14-hotfix.
Preamble: pipeline-preamble.md
Sessions: sessions-reference.md — requires active_session unless waived; reports under docs/sessions/{id}/reports/.
Cross-cutting: considerations.md, deployment-catalog.md, connectivity-gates.md.
State agent: project .cursor/agents/workflow-state-manager.md if present; else edit workflow-state.yaml per workflow-state-reference.md
Connectivity (stage 15)
When a browser frontend exists, default depth: H4–H5 before H3 for UI-related reports.
See connectivity-gates for tier definitions.
User is source of truth. AskQuestion sets infra depth, health tier (H0–H6), target URL, and
whether to run costly full E2E suites.
Session management
Per sessions-reference.md §10 and workflow-state-agent-protocol.md.
- Agent
read_context must return active_session (or blocking deviation).
- Current stage must appear in
active_session.routing_plan unless user amends plan.
- Write stage reports to
active_session.artifacts_dir/reports/ when this stage produces a report.
- On completion: update routing-plan entry status; mirror
project.stages.{key} via agent update.
- 00-context exempt from active_session requirement (session opener).
Report:
reports/service-health.md.
State management
Agent protocol: workflow-state-agent-protocol.md.
Stage key: stages.15-service-health.
Invoke workflow-state-manager read_context before any other action; update after each
substep. Do not edit workflow-state.yaml directly.
Per-run reports under docs/service-health-reports/ (ephemeral).
Two-layer model
| Layer | Question | Pass when |
|---|
| Infra | Is the right thing deployed and wired? | Health 200, DB migrated, secrets present |
| Behavior | Do primary API flows work end-to-end? | Approved smokes return expected status/body |
Record Infra overall, E2E overall, and Overall separately in the report.
Health ≠ primary-journey-ready
Liveness//health and dependency probes can be green while the primary user journey fails
(hang, timeout, wrong runtime).
- If H3 is in scope: H3 fail or hang ⇒ Overall FAIL (and Behavior FAIL) even when H1 is PASS.
- Do not report Overall PASS on infra-only evidence when the requested depth includes H3.
- On H3 P0 during an active evolve cycle: recommend pause 16-evolve → 14-hotfix.
Single-env / live role
If only one deployed stack exists (env_role: staging_as_live), label checks and reports as
live/prod — do not imply a safer non-prod target.
TAC-to-IWXXM (ADR-034 / EV-044): Dual DOKS clusters — use env_role: staging vs
prod and the matching hosts (*.staging.tac-to-iwxxm.com / staging LB vs
api|app.tac-to-iwxxm.com). Do not use sole-stack language when cluster
metar-iwxxm-staging is provisioned. Promotions to main require Staging smoke green.
Health tiers (default recommendations)
| Tier | Scope | Example |
|---|
| H0 | Local integration | uv run pytest tests/integration -v |
| H0ci | GitHub main CI | Latest required workflow on main → success |
| H1 | Liveness | GET {{HEALTH_ENDPOINT}} → 200 |
| H2 | DB ready | migrations at head; pool connects |
| H3 | API smoke | documented curl/pytest against {{STAGING_URL}} |
| H4 | Browser CORS | connectivity pytest or verify_connectivity.sh |
| H5 | Frontend bundle | VITE_* wiring check (when frontend exists) |
| H6 | Full UJ suite | pytest tests/e2e/ -m live or browser automation |
| Trigger | Recommend infra | Recommend behavior |
|---|
| Routine | H1 + H2 + H0ci (advisory) | H3 |
| Post-deploy / post-hotfix | H1–H2 + deploy metadata + H0ci (blocking) | H3 + H4–H5 if UI |
User-reported CI failure on main | H0ci first | H3 only if CI green |
| User-reported “UI broken / Failed to fetch” | H4–H5 first | H3 |
| Weekly deep | H1–H2 + backlog metrics | H6 (explicit approval) |
Never auto-run H6 without AskQuestion approval.
Main CI (H0ci)
gh run list --branch main --workflow ci-cd.yml --limit 5
gh run view <run-id> --json conclusion,status,headSha,url,workflowName
Pass when conclusion is success for the run matching the SHA under investigation.
Record run URL in the service-health report.
Update workflow-state.yaml §deployment.staging.health_tiers.h0ci_github_main when H0ci runs.
Test-driven investigation
Same as bug-investigation: repro test first, confirm with user,
then production checks.
Delta / feature-addition mode
If user request is feature addition: recommend 16-evolve instead.
After feature deploy, optional health pass scoped to new Fn journeys.
Workflow (summary)
Phase 0 — Interview
- Target environment (local / staging / production)
- Base URL
{{STAGING_URL}}, DB reachability (read-only ok?)
- Depth: infra only vs include H3/H4
- Known symptoms, recent deploys, failing UJ-NNN
Phase 1 — Infra checks
- Deploy revision matches expected git SHA / image tag
- H0ci when post-deploy/post-hotfix (blocking unless waived)
GET {{HEALTH_ENDPOINT}}- Migration status (
{{MIGRATION_COMMAND}} or platform equivalent)
- Required env vars present (names only in report, never values)
Phase 2 — Behavior checks
- Run approved H3 script from
docs/user-journeys.md or docs/deploy.md §Runbook
- If UI involved: H4–H5 per connectivity-gates
- Capture latency (informational)
Phase 3 — Report & route
Write docs/service-health-reports/YYYY-MM-DD-[slug].md with remediation routing:
none | config | data reseed | fix main CI | 14-hotfix (code).
Update workflow-state.yaml §stages.15-service-health.
Output rules
- No secret values in reports.
- Infra fail → do not claim API broken without evidence.
- Code defects → bug report before hotfix.
- Compare live routes to
docs/api-contract.md when applicable.
- Do not mark Overall PASS when main CI is red after recent merge/hotfix unless user waives.
