catalog-audit

# Catalog audit Audit catalog entries that are **already in the registry** for three kinds of drift: 1. **Still valid** - the pinned OCI image still resolves; `repository.url` still loads. 2. **Still active** - the upstream repo isn't archived/disabled and isn't stale. 3. **Still canonically correct** - declared env vars and the network-permission profile match what the upstream project actually documents and requires. This is **report-only**. Produce findings for a human to triage; never edit a `server.json`, open a PR, or change permissions. The output is a dated markdown report under `audits/`. Checks 1 and 2 are mechanical and run via the bundled script. Check 3 needs judgment over upstream docs and is done by fanning out sub-agents. Grading rules and severity live in [references/audit-criteria.md](references/audit-criteria.md) - read it before writing findings. ## Step 1 - resolve scope Infer scope from the request, don't ask for rigid flags. Map intent to a server set: - "the community servers" / "community tier" -> `--tier community` - "official servers" -> `--tier official` - "the whole catalog" / "everything" -> `--tier all` - "re-check X, Y, Z" / a list of names -> `--names X,Y,Z` - a specific registry tree -> `--registry toolhive|official|all` (default `toolhive`) The repo has two server trees: `registries/toolhive/servers/` (contribution path, default) and `registries/official/servers/` (curated subset). Default to `toolhive` unless the user clearly means otherwise. Skills (`registries/toolhive/skills/`) are **out of scope** unless explicitly asked. If asked, audit them separately (their `packages[]` carry `registryType: git` with a `ref`/`subfolder` and `registryType: oci` images - check the git ref resolves and the repo is active; there are no env vars or network permissions to canonicalize). State the resolved scope and the entry count back to the user before crawling, so a mistaken scope is caught early. ## Step 2 - mechanical checks (script) Run the bundled deterministic script. It needs `gh` (authenticated) and `docker` or `skopeo` on PATH. ```bash python3 .claude/skills/catalog-audit/scripts/check_entries.py \ --tier community \ --out <scratch>/mechanical.json ``` Use `--names a,b,c` for a subset, or `--paths .../server.json ...` for exact files. The script prints a JSON array (also written to `--out`): per entry it reports `imageExists` / `imageMethod`, `repoResolves`, `archived`, `disabled`, `lastPushISO`, `monthsStale` / `isStale`, `openIssues`, plus `dockyard` and `tier`. `null` on an image/repo field means inconclusive (tooling couldn't run / transient), **not** a pass. It also reports `officialDrift`: for an Official-tier entry, whether its `registries/official` mirror has drifted from the `registries/toolhive` source - `false` (in sync), a list of differing top-level keys (e.g. `["repository"]`), or `null` (no mirror, e.g. Community-tier). A non-empty list is a finding worth reporting and means a fix needs applying to both copies. ## Step 3 - deep canonical-correctness checks (sub-agents) For each in-scope entry, spawn a sub-agent to do check 3. Keep it cheap: these are independent, doc-reading tasks. For a handful of entries, launch `Task` agents in parallel (batches of ~8). For a large sweep (dozens), prefer the `Workflow` tool to pipeline the fan-out if the user has opted into it; otherwise batch `Task` agents. Give each sub-agent: - the `server.json` path and its parsed env vars + `permissions` block, - the `repository.url` (and note if `dockyard:true`, so it reads the upstream repo, not the dockyard wrapper), - the relevant criteria from [references/audit-criteria.md](references/audit-criteria.md). Instruct each agent to read the upstream README and relevant source (via `gh api repos/{owner}/{repo}/readme`, `gh api`/`get_file_contents` for files, `search_code` for env-var references, or `WebFetch`), then return **structured findings only**: ```json { "name": "<server>", "envVars": [{"name": "...", "issue": "...", "severity": "...", "evidence": "<url-or-file:line>"}], "permissions": [{"issue": "...", "severity": "...", "evidence": "..."}], "couldNotVerify": ["..."] } ``` Each finding must cite upstream evidence (a URL or `file:line`). Findings without evidence are not actionable - drop them or mark `couldNotVerify`. ## Step 4 - verify findings before reporting (adversarial) Single-pass deep checks over-report - this is not optional cleanup, it's a core step. A cheap finder reliably flags plausible-but-wrong issues: a host that a leading-dot `allow_host` already covers, an "unused" var its own evidence shows is used, an `insecure_allow_all` that is actually justified, a 401 from a healthy OAuth-gated remote. For every high/critical finding (and ideally all of them), spawn an **independent agent - a stronger model than the finder** - to re-check it against upstream, prompted to refute by default: "confirm only if you can re-derive this from upstream; mark refuted if the entry is actually correct; uncertain if you can't tell." Drop refuted findings; keep confirmed ones with the verifier's note. In the first official-tier run this stage refuted ~6 high and ~21 medium false positives that had survived the finder. When fixes are later applied, re-verify each one against upstream while editing (grep the actual var name / host) - a cheap third check that catches the cases a verifier still misses (e.g. a contested host the second pass got wrong). ## Step 5 - merge and write the report Merge mechanical results (step 2) with deep findings (step 3) into severity-ranked findings using the scale in the criteria file. Write to `audits/<scope>-<YYYY-MM-DD>.md` (create `audits/` if absent; get the date with `date +%F`). Use this structure: ```markdown # Catalog audit - <scope> - <date> Scope: <description> (<N> entries). Methods: docker/skopeo image probe, gh repo API, sub-agent upstream-doc review. Report only - no entries were modified. ## Summary | entry | image | repo | active | env issues | perm issues | worst | |-------|-------|------|--------|-----------|-------------|-------| ... ## Critical ... ## High ... ## Medium ... ## Low ... ## Could not verify <entries/fields where a check was inconclusive> ## Clean <entries with no findings> ``` Each finding lists the entry, category, detail, evidence (URL/command), and a suggested action for the human (since this is report-only). Lead with the worst. ## Step 6 - hand off Show the user the summary table and the top findings inline. Do **not** commit the report or edit any entry unless the user explicitly asks. If they want fixes, that's a separate pass (and `mcp-review` / `add-mcp-server` cover the editing rules). When fixing an Official-tier entry, remember it has a mirror copy (see Notes) that must be updated too. ## Notes - **Inconclusive is not clean.** Surface `null` image/repo results and unreachable upstream docs as "could not verify," never as passing. - **Check wildcard coverage before flagging a missing host.** `allow_host` is Squid-style: a leading-dot entry (`.example.com`) already matches every subdomain. List the actual `allow_host` and confirm the needed host isn't already covered before reporting it missing. (Both the finder and a verifier missed this on `aws-documentation` in the first run.) - **A remote endpoint returning 401/403 is alive,** not a finding - it's auth/OAuth-gated. Only DNS failure, connection refused, 404, 5xx, or TLS errors mean it may be down. - **Official-tier entries are mirrored in `registries/official/servers/`** (a copy, not a symlink). Flag when an entry's `registries/official` copy has drifted from its `registries/toolhive` source - that drift is itself a finding. And any fix to an Official-tier entry must be applied to both copies (CI reconciles only the auto-populated `metadata` / `tool_definitions`, not the hand-edited fields). - **Be fair on staleness.** A small, finished server can be quiet for good reasons; weigh `monthsStale` against the server's scope before flagging. - **Be fair on `insecure_allow_all`.** Servers that reach arbitrary user-supplied hosts (fetch, scrapers, browsers, db/cluster/registry clients) may legitimately need it; flag it only where a concrete scoped host list is feasible, and propose that list.