sync-docs

name	sync-docs
description	Use when the user asks to update, sync, or refresh the Hawk docs (quickstart, configuration reference, infrastructure description and diagrams) to reflect changes since the last documentation sync. Triggers on "update the docs", "sync docs", "docs are stale", "refresh quickstart", "update the architecture diagram", or after merging infra/config/CLI changes that affect user-visible setup.

Sync Hawk docs to current state

Bring the user-facing documentation back in line with the code. The focus areas, in priority order, are:

1. Quickstart — docs/getting-started/index.md, docs/getting-started/installation.md
2. Configuration reference — docs/getting-started/configuration.md, Pulumi.example.yaml
3. Infrastructure description & diagram — docs/infrastructure/architecture.md (mermaid diagrams), docs/infrastructure/middleman.md, docs/infrastructure/security.md, docs/infrastructure/deployment.md, docs/infrastructure/new-environment.md, docs/infrastructure/database.md, docs/infrastructure/managing.md
4. Top-level READMEs — README.md, hawk/README.md, infra/README.md, AGENTS.md

Only touch what actually changed. Drift-by-drift, not a rewrite.

Step 0: Pick a mode — since-baseline diff or comprehensive audit

Two modes, based on how stale the docs look:

Mode	When to use	What it catches
Since-baseline diff (Steps 1–3)	Last doc-sync commit is recent (days/weeks) and the user asks to "refresh" / "sync"	Drift introduced by recent merges. Fast. Misses accumulated drift that pre-dates the baseline.
Comprehensive audit (Step 1.5)	Docs haven't been touched in months, user says "docs are stale", asks for a "comprehensive review", or specifically asks to add a component (which usually means other related drift has accumulated too)	Everything: missing config keys, renamed components, missing diagram nodes, broken env-var tables. Slower.

If unsure, ask the user — but default to comprehensive audit when the user uses words like "stale", "haven't been updated in a while", "comprehensive", or names a specific missing component. The since-baseline diff will report "nothing to do" in those cases and be wrong about it.

Step 1: Establish the baseline (last doc-sync commit)

Find the most recent commit whose intent was to bring docs in sync with code:

git log --grep='update docs to describe current state' -i --oneline -n 5
git log --grep='^docs:' --oneline -n 10
git log --oneline -- README.md docs/ | head -20

The conventional message is update docs to describe current state (#NNN). Pick the most recent one — call its SHA $BASE. If the user names a different reference point (PR number, tag, "last week"), use that instead.

If no such commit exists or the answer is ambiguous, ask the user which point to diff against before continuing. Do not guess.

Step 2: Survey what changed since $BASE

Pull a structured view of changes that could plausibly affect docs:

git log --oneline $BASE..HEAD
git diff --stat $BASE..HEAD -- \
  Pulumi.example.yaml Pulumi.yaml \
  infra/ \
  scripts/dev/new-dev-env.sh scripts/dev/*.sh scripts/dev/*.py \
  hawk/hawk/cli/ hawk/hawk/api/config.py hawk/hawk/api/auth.py \
  .nvmrc package.json pyproject.toml hawk/pyproject.toml \
  AGENTS.md AGENTS.local.example.md

For each file in that list, run git diff $BASE..HEAD -- <file> and read it. Note that signals do not auto-translate to doc edits — interpret them.

Step 1.5: Comprehensive audit (replace Step 2 in audit mode)

Skip if you chose since-baseline diff in Step 0. Run all three of these in parallel; they're independent and each fully discovers its own surface:

1. Pulumi config keys — run the local linter to enumerate every key the code reads and which keys are undocumented:

   python3 scripts/ops/check-config-docs.py
   

   Exit code 1 means there are undocumented keys; the report lists them. Decide per-key: document in Pulumi.example.yaml + docs/getting-started/configuration.md, or allowlist in the script with a one-line reason (only for genuinely internal keys — dev-env bootstrap, METR-specific, shared-infra companion fields).

2. CLI env vars — grep -rEn 'os\.(environ|getenv)|HAWK_[A-Z_]+' hawk/hawk/cli/ hawk/hawk/core/cli_config*.py 2>/dev/null and cross-check the env-var tables in docs/getting-started/installation.md and hawk/README.md. Add missing rows; remove or clarify ones the CLI doesn't read.

3. Architecture components — list every top-level entry in infra/__main__.py and every service module under infra/hawk/ / infra/core/. For each, confirm it appears in the mermaid diagrams under docs/infrastructure/architecture.md with the correct deployment target. To avoid guessing the target:

   grep -rEn 'DockerLambda|aws\.lambda_|aws\.ecs|aws\.batch|aws\.cloudfront|pulumi_kubernetes' infra/hawk/ infra/core/ | head -40
   

   That tells you whether something is Lambda, ECS Fargate, AWS Batch, CloudFront, or EKS-resident. Don't recall it — read it.

The output of these three audits is the work list. Apply Steps 3–8 to it.

Step 3: Map code changes to doc sections

Use this table to decide which docs to update. A single code change can fan out to multiple doc files.

Code change	Doc(s) to update
New / renamed / removed Pulumi config key in infra/core/__init__.py or infra/__main__.py	Pulumi.example.yaml (add commented entry with default + description), docs/getting-started/configuration.md (add row in the right table)
Pulumi.example.yaml itself edited	Cross-check docs/getting-started/configuration.md tables; quickstart steps that reference specific keys
scripts/dev/new-dev-env.sh changed flow	docs/infrastructure/new-environment.md, AGENTS.md "Dev environments" section
Prerequisites changed (.nvmrc, pyproject.toml Python version, new system dep, new brew install need)	docs/getting-started/index.md step 1, docs/getting-started/installation.md Prerequisites
New CLI subcommand / flag / env var in hawk/hawk/cli/	docs/getting-started/installation.md env var table, docs/user-guide/cli-reference.md, README CLI sections
OIDC / auth flow changes (hawk/hawk/api/auth.py, infra/.../auth*)	docs/getting-started/configuration.md Authentication section, docs/infrastructure/security.md
New top-level infra component (new directory under infra/, new service in infra/__main__.py, new Helm chart)	docs/infrastructure/architecture.md — both mermaid diagrams + prose; possibly a new file under docs/infrastructure/
Component renamed / removed / moved (e.g. token-broker, middleman, runner)	Every mermaid diagram referencing the old name, plus prose mentions across docs/infrastructure/ and READMEs
Network / VPC / subnet topology change	docs/infrastructure/architecture.md "Detailed Architecture" diagram, docs/infrastructure/security.md
Database schema / engine / failover changes	docs/infrastructure/database.md
Datadog / Sentry / OTel / X-Ray wiring change	docs/infrastructure/architecture.md observability nodes, AGENTS.md "Observability" section
Region or AZ constraints change	"Do not deploy to us-east-1" warning in docs/getting-started/index.md and docs/infrastructure/new-environment.md

If a change does not map to any row, default to leave the docs alone. Don't invent doc churn.

Step 4: Inventory and verify mermaid diagrams

Mermaid diagrams live in:

grep -rln '```mermaid' docs/ README.md

Currently: docs/infrastructure/architecture.md (overview + detailed), docs/infrastructure/security.md. Each diagram has the same components named consistently — when a component is added, renamed, or removed in code, update every diagram that mentions it. Don't update one and leave the others.

Sanity-check the names against reality before editing the diagram:

ls infra/                              # top-level components
grep -rE '^[a-z_]+ = ' infra/__main__.py | head -40   # resource bindings

After editing a mermaid block, run the local validator:

( cd scripts/ops/check-mermaid && npm ci && node check.mjs )

It parses every block under docs/ via mermaid + jsdom and exits non-zero on any syntax error. (CI runs the same script on PRs.) A broken diagram silently degrades the docs site — fail fast locally.

Step 5: Update the configuration reference cleanly

docs/getting-started/configuration.md has multiple tables (Required Settings, Domain & DNS, Authentication, Authentication (advanced), Infrastructure Options, Observability, Optional Integrations). When adding a new config key:

• Put it in the right table — Required vs. Optional matters, and observability/auth keys belong in their dedicated sections, not under Infrastructure Options
• Cross-add it as a commented entry in Pulumi.example.yaml with # default: X and a one-line purpose
• Read the default from the code, don't recall it from memory. Most keys are read in infra/lib/config.py with a pattern like cfg.get("keyName") or "<default>" or cfg.get_bool("keyName") or False. The literal after or IS the default — copy it verbatim into both the doc table and the Pulumi.example.yaml comment. (Bad defaults get caught by Copilot review eventually, but it's faster to be right the first time.)
• If it has a non-obvious gotcha, add an !!! warning or !!! tip block — short

After editing, run the linter to confirm the new key is now considered documented:

python3 scripts/ops/check-config-docs.py

When removing a key, search for stragglers:

grep -rn 'old_key_name' docs/ Pulumi.example.yaml infra/ README.md

Step 6: Update the quickstart

docs/getting-started/index.md is a numbered walkthrough. Constraints:

• Each numbered step must still produce a working state when run literally — no aspirational instructions
• The brew install line, nvm install, and pulumi/uv versions should match .nvmrc and pyproject.toml
• The us-east-1 warning stays until both root causes (referenced in the warning) are actually fixed upstream — check infra/core/__init__.py if unsure

Run through the numbered steps mentally as a brand-new user before considering the edit done.

Step 7: Update READMEs only for changes a reader would notice in 30s

README.md, hawk/README.md, infra/README.md, and AGENTS.md should reflect:

• What the project / subdirectory is (only if scope changed)
• Top-level setup commands users will run first
• Dev-environment workflow if new-dev-env.sh or seed changed

Skip cosmetic rewrites. If the existing text is accurate, leave it.

Step 8: Review the resulting diff and commit

git diff --stat
git diff -- docs/ README.md Pulumi.example.yaml AGENTS.md

Read every changed hunk. Watch for:

• Stale terminology (e.g., "Terraform" — Terraform was removed; everything is Pulumi now)
• Broken internal links (docs/foo/bar.md paths)
• Mermaid blocks that no longer parse
• Tables where a column heading shifted but rows didn't

Splitting into commits

If the work spans more than just doc text — e.g. you also touched CI, scripts, or the skill itself — split commits by logical area rather than dumping everything in one. A clean shape is:

1. docs: update to describe current state — the user-facing doc/config edits
2. chore: ... or ci: ... — any supporting infrastructure changes (scripts, workflows, gitignore tweaks)

This makes Copilot reviews easier to read and lets future runs of this skill find the right baseline.

The doc commit message itself is the baseline marker that Step 1 greps for — keep the docs: update to describe current state phrasing so future invocations find it cleanly. No tag or annotation needed.

Open a draft PR (per global preference) and add Copilot as a reviewer.

What to avoid

• Don't rewrite stable prose to match a personal style. Edit only what's inaccurate.
• Don't add forward-looking documentation for changes that aren't on main yet.
• Don't auto-update Pulumi.stg.yaml or Pulumi.prd.yaml — they're symlinks into a separate private config repo (METR/hawk-config) and shouldn't drift from real deployed state.
• Don't touch docs/superpowers/ or hawk/docs/superpowers/ — those are agent specs/plans, not user docs.
• Don't touch docs/roadmap.md unless the user asks; it's editorial, not derived from code.
• Don't introduce comments or summaries inside doc files that describe the edit (e.g. "Updated for #412") — git history is the changelog.
• Don't write implementation-detail constants into architecture prose — Lambda memory sizes, timeout values, CPU/RAM allocations, replica counts, retry attempts, ACU sizing, etc. These rot the moment someone tunes them and add zero value to a reader trying to understand the system. Document them only in the config reference (where the constant is the contract for what pulumi up produces) or in component-specific runbooks if anywhere. Conceptual specifics (e.g. S3 path layout like evals/<job_id>/*, named components, request flows) are fine — those describe the architecture, not its sizing.