sync-docs

# Sync Hawk docs to current state Bring the user-facing documentation back in line with the code. The focus areas, in priority order, are: 1. **Quickstart** — `docs/getting-started/index.md`, `docs/getting-started/installation.md` 2. **Configuration reference** — `docs/getting-started/configuration.md`, `Pulumi.example.yaml` 3. **Infrastructure description & diagram** — `docs/infrastructure/architecture.md` (mermaid diagrams), `docs/infrastructure/middleman.md`, `docs/infrastructure/security.md`, `docs/infrastructure/deployment.md`, `docs/infrastructure/new-environment.md`, `docs/infrastructure/database.md`, `docs/infrastructure/managing.md` 4. **Top-level READMEs** — `README.md`, `hawk/README.md`, `infra/README.md`, `AGENTS.md` Only touch what actually changed. Drift-by-drift, not a rewrite. ## Step 0: Pick a mode — since-baseline diff or comprehensive audit Two modes, based on how stale the docs look: | Mode | When to use | What it catches | |---|---|---| | **Since-baseline diff** (Steps 1–3) | Last doc-sync commit is recent (days/weeks) and the user asks to "refresh" / "sync" | Drift introduced by recent merges. Fast. Misses accumulated drift that pre-dates the baseline. | | **Comprehensive audit** (Step 1.5) | Docs haven't been touched in months, user says "docs are stale", asks for a "comprehensive review", or specifically asks to add a component (which usually means other related drift has accumulated too) | Everything: missing config keys, renamed components, missing diagram nodes, broken env-var tables. Slower. | If unsure, ask the user — but default to **comprehensive audit** when the user uses words like "stale", "haven't been updated in a while", "comprehensive", or names a specific missing component. The since-baseline diff will report "nothing to do" in those cases and be wrong about it. ## Step 1: Establish the baseline (last doc-sync commit) Find the most recent commit whose intent was to bring docs in sync with code: ```bash git log --grep='update docs to describe current state' -i --oneline -n 5 git log --grep='^docs:' --oneline -n 10 git log --oneline -- README.md docs/ | head -20 ``` The conventional message is `update docs to describe current state (#NNN)`. Pick the most recent one — call its SHA `$BASE`. If the user names a different reference point (PR number, tag, "last week"), use that instead. If no such commit exists or the answer is ambiguous, ask the user which point to diff against before continuing. Do not guess. ## Step 2: Survey what changed since `$BASE` Pull a structured view of changes that could plausibly affect docs: ```bash git log --oneline $BASE..HEAD git diff --stat $BASE..HEAD -- \ Pulumi.example.yaml Pulumi.yaml \ infra/ \ scripts/dev/new-dev-env.sh scripts/dev/*.sh scripts/dev/*.py \ hawk/hawk/cli/ hawk/hawk/api/config.py hawk/hawk/api/auth.py \ .nvmrc package.json pyproject.toml hawk/pyproject.toml \ AGENTS.md AGENTS.local.example.md ``` For each file in that list, run `git diff $BASE..HEAD -- <file>` and read it. Note that signals do not auto-translate to doc edits — interpret them. ## Step 1.5: Comprehensive audit (replace Step 2 in audit mode) Skip if you chose since-baseline diff in Step 0. Run all three of these in parallel; they're independent and each fully discovers its own surface: 1. **Pulumi config keys** — run the local linter to enumerate every key the code reads and which keys are undocumented: ```bash python3 scripts/ops/check-config-docs.py ``` Exit code 1 means there are undocumented keys; the report lists them. Decide per-key: document in `Pulumi.example.yaml` + `docs/getting-started/configuration.md`, or allowlist in the script with a one-line reason (only for genuinely internal keys — dev-env bootstrap, METR-specific, shared-infra companion fields). 2. **CLI env vars** — `grep -rEn 'os\.(environ|getenv)|HAWK_[A-Z_]+' hawk/hawk/cli/ hawk/hawk/core/cli_config*.py 2>/dev/null` and cross-check the env-var tables in `docs/getting-started/installation.md` and `hawk/README.md`. Add missing rows; remove or clarify ones the CLI doesn't read. 3. **Architecture components** — list every top-level entry in `infra/__main__.py` and every service module under `infra/hawk/` / `infra/core/`. For each, confirm it appears in the mermaid diagrams under `docs/infrastructure/architecture.md` with the correct deployment target. To avoid guessing the target: ```bash grep -rEn 'DockerLambda|aws\.lambda_|aws\.ecs|aws\.batch|aws\.cloudfront|pulumi_kubernetes' infra/hawk/ infra/core/ | head -40 ``` That tells you whether something is Lambda, ECS Fargate, AWS Batch, CloudFront, or EKS-resident. Don't recall it — read it. The output of these three audits is the work list. Apply Steps 3–8 to it. ## Step 3: Map code changes to doc sections Use this table to decide which docs to update. A single code change can fan out to multiple doc files. | Code change | Doc(s) to update | |---|---| | New / renamed / removed Pulumi config key in `infra/core/__init__.py` or `infra/__main__.py` | `Pulumi.example.yaml` (add commented entry with default + description), `docs/getting-started/configuration.md` (add row in the right table) | | `Pulumi.example.yaml` itself edited | Cross-check `docs/getting-started/configuration.md` tables; quickstart steps that reference specific keys | | `scripts/dev/new-dev-env.sh` changed flow | `docs/infrastructure/new-environment.md`, AGENTS.md "Dev environments" section | | Prerequisites changed (`.nvmrc`, `pyproject.toml` Python version, new system dep, new `brew install` need) | `docs/getting-started/index.md` step 1, `docs/getting-started/installation.md` Prerequisites | | New CLI subcommand / flag / env var in `hawk/hawk/cli/` | `docs/getting-started/installation.md` env var table, `docs/user-guide/cli-reference.md`, README CLI sections | | OIDC / auth flow changes (`hawk/hawk/api/auth.py`, `infra/.../auth*`) | `docs/getting-started/configuration.md` Authentication section, `docs/infrastructure/security.md` | | New top-level infra component (new directory under `infra/`, new service in `infra/__main__.py`, new Helm chart) | `docs/infrastructure/architecture.md` — **both** mermaid diagrams + prose; possibly a new file under `docs/infrastructure/` | | Component renamed / removed / moved (e.g. token-broker, middleman, runner) | Every mermaid diagram referencing the old name, plus prose mentions across `docs/infrastructure/` and READMEs | | Network / VPC / subnet topology change | `docs/infrastructure/architecture.md` "Detailed Architecture" diagram, `docs/infrastructure/security.md` | | Database schema / engine / failover changes | `docs/infrastructure/database.md` | | Datadog / Sentry / OTel / X-Ray wiring change | `docs/infrastructure/architecture.md` observability nodes, AGENTS.md "Observability" section | | Region or AZ constraints change | "Do not deploy to `us-east-1`" warning in `docs/getting-started/index.md` and `docs/infrastructure/new-environment.md` | If a change does not map to any row, default to **leave the docs alone**. Don't invent doc churn. ## Step 4: Inventory and verify mermaid diagrams Mermaid diagrams live in: ```bash grep -rln '```mermaid' docs/ README.md ``` Currently: `docs/infrastructure/architecture.md` (overview + detailed), `docs/infrastructure/security.md`. Each diagram has the same components named consistently — when a component is added, renamed, or removed in code, update **every** diagram that mentions it. Don't update one and leave the others. Sanity-check the names against reality before editing the diagram: ```bash ls infra/ # top-level components grep -rE '^[a-z_]+ = ' infra/__main__.py | head -40 # resource bindings ``` After editing a mermaid block, run the local validator: ```bash ( cd scripts/ops/check-mermaid && npm ci && node check.mjs ) ``` It parses every block under `docs/` via `mermaid` + `jsdom` and exits non-zero on any syntax error. (CI runs the same script on PRs.) A broken diagram silently degrades the docs site — fail fast locally. ## Step 5: Update the configuration reference cleanly `docs/getting-started/configuration.md` has multiple tables (Required Settings, Domain & DNS, Authentication, Authentication (advanced), Infrastructure Options, Observability, Optional Integrations). When adding a new config key: - Put it in the **right** table — Required vs. Optional matters, and observability/auth keys belong in their dedicated sections, not under Infrastructure Options - Cross-add it as a commented entry in `Pulumi.example.yaml` with `# default: X` and a one-line purpose - **Read the default from the code, don't recall it from memory.** Most keys are read in `infra/lib/config.py` with a pattern like `cfg.get("keyName") or "<default>"` or `cfg.get_bool("keyName") or False`. The literal after `or` IS the default — copy it verbatim into both the doc table and the `Pulumi.example.yaml` comment. (Bad defaults get caught by Copilot review eventually, but it's faster to be right the first time.) - If it has a non-obvious gotcha, add an `!!! warning` or `!!! tip` block — short After editing, run the linter to confirm the new key is now considered documented: ```bash python3 scripts/ops/check-config-docs.py ``` When removing a key, search for stragglers: ```bash grep -rn 'old_key_name' docs/ Pulumi.example.yaml infra/ README.md ``` ## Step 6: Update the quickstart `docs/getting-started/index.md` is a numbered walkthrough. Constraints: - Each numbered step must still produce a working state when run literally — no aspirational instructions - The `brew install` line, `nvm install`, and `pulumi`/`uv` versions should match `.nvmrc` and `pyproject.toml` - The us-east-1 warning stays until both root causes (referenced in the warning) are actually fixed upstream — check `infra/core/__init__.py` if unsure Run through the numbered steps mentally as a brand-new user before considering the edit done. ## Step 7: Update READMEs only for changes a reader would notice in 30s `README.md`, `hawk/README.md`, `infra/README.md`, and `AGENTS.md` should reflect: - What the project / subdirectory **is** (only if scope changed) - Top-level setup commands users will run first - Dev-environment workflow if `new-dev-env.sh` or `seed` changed Skip cosmetic rewrites. If the existing text is accurate, leave it. ## Step 8: Review the resulting diff and commit ```bash git diff --stat git diff -- docs/ README.md Pulumi.example.yaml AGENTS.md ``` Read every changed hunk. Watch for: - Stale terminology (e.g., "Terraform" — Terraform was removed; everything is Pulumi now) - Broken internal links (`docs/foo/bar.md` paths) - Mermaid blocks that no longer parse - Tables where a column heading shifted but rows didn't ### Splitting into commits If the work spans more than just doc text — e.g. you also touched CI, scripts, or the skill itself — split commits by logical area rather than dumping everything in one. A clean shape is: 1. `docs: update to describe current state` — the user-facing doc/config edits 2. `chore: ...` or `ci: ...` — any supporting infrastructure changes (scripts, workflows, gitignore tweaks) This makes Copilot reviews easier to read and lets future runs of this skill find the right baseline. The doc commit message itself **is** the baseline marker that Step 1 greps for — keep the `docs: update to describe current state` phrasing so future invocations find it cleanly. No tag or annotation needed. Open a **draft** PR (per global preference) and add Copilot as a reviewer. ## What to avoid - **Don't rewrite stable prose** to match a personal style. Edit only what's inaccurate. - **Don't add forward-looking documentation** for changes that aren't on `main` yet. - **Don't auto-update `Pulumi.stg.yaml` or `Pulumi.prd.yaml`** — they're symlinks into a separate private config repo (`METR/hawk-config`) and shouldn't drift from real deployed state. - **Don't touch `docs/superpowers/` or `hawk/docs/superpowers/`** — those are agent specs/plans, not user docs. - **Don't touch `docs/roadmap.md`** unless the user asks; it's editorial, not derived from code. - **Don't introduce comments or summaries inside doc files** that describe the edit (e.g. "Updated for #412") — git history is the changelog. - **Don't write implementation-detail constants into architecture prose** — Lambda memory sizes, timeout values, CPU/RAM allocations, replica counts, retry attempts, ACU sizing, etc. These rot the moment someone tunes them and add zero value to a reader trying to understand the system. Document them only in the **config reference** (where the constant *is* the contract for what `pulumi up` produces) or in component-specific runbooks if anywhere. Conceptual specifics (e.g. S3 path layout like `evals/<job_id>/*`, named components, request flows) are fine — those describe the architecture, not its sizing.