verify-upstream-config-examples

name	verify-upstream-config-examples
description	Detect drift between testagent's upstream config corpus and vendor docs; surface net-new examples as a draft PR.
triggers	["verify upstream config examples","check upstream config drift","verify config corpus"]

verify-upstream-config-examples

Detects drift between the vendored corpus under testdata/upstream-examples/ and live vendor docs. Net-new examples that pass testagent <vendor> validate --strict are written to the corpus and surfaced as a draft PR.

Vendor dispatch

Phrase	Vendor arg
"verify upstream config examples" (no vendor named)	ask the user or default to all
"verify claude config corpus" / "check claude drift"	claude
"verify codex config corpus" / "check codex drift"	codex
"verify cursor config corpus" / "check cursor drift"	cursor
"verify upstream config examples all"	all (runs all three sequentially)

Workflow

1. Scope check

Vendor must be claude, codex, cursor, or all. If ambiguous, ask: "Which vendor — claude, codex, cursor, or all?"

2. Fetch the vendor's source

Per sources.md:

• claude: curl -sL https://code.claude.com/llms.txt — produces a flat list of doc URLs. Grep for lines containing hooks or mcp; curl each linked page (also markdown).
• cursor: curl -sL https://cursor.com/llms.txt — same shape. Linked pages under /docs/hooks and /docs/context/mcp.
• codex: gh api repos/openai/codex/contents/README.md -H 'Accept: application/vnd.github.raw' — inline fenced blocks; no link-following needed.

3. Extract fenced code blocks

From each fetched page, extract all fenced code blocks of the relevant language:

• claude and cursor: ```json blocks.
• codex: ```toml blocks.

Use awk to detect ```json / ```toml openers and matching ``` closers. Produce one candidate file per block.

4. Diff against local corpus

For each candidate block:

1. Compute SHA256 of the block content.
2. Compare against SHA256 of every file in testdata/upstream-examples/<vendor>/.
3. Skip blocks already vendored verbatim (exact content match).
4. Flag blocks where a same-name fixture exists but content differs (drift in an existing fixture).

5. Curate net-new blocks

For each net-new or drifted block, run:

/tmp/testagent <vendor> validate --strict < <candidate-file>

• Passes: propose for corpus inclusion (write to testdata/upstream-examples/<vendor>/).
• Fails: surface as a separate update-compatibility follow-up item. Do NOT vendor the failing block. Include it in the ### Skipped — fail --strict section of the PR body.

6. De-dupe check

Before opening any PR, run:

gh pr list --search 'in:title "upstream-config drift detected (<vendor>)" state:open'

If a matching open PR exists, skip PR creation and report the existing PR URL.

7. PR creation

If net-new passing fixtures were found:

1. Write each fixture to testdata/upstream-examples/<vendor>/<slug>.{json,toml}.
2. Write a .source sibling with:

   url: <source-page-url>
   verified: <YYYY-MM-DD>
   

3. Run testagent <vendor> validate --strict against every newly written fixture to confirm.
4. Create a branch: chore/upstream-drift-<vendor>-<date>.
5. Commit: chore(testdata): upstream-config drift detected (<vendor>).
6. Open a draft PR:

   gh pr create --draft \
     --title "chore(testdata): upstream-config drift detected (<vendor>)" \
     --body "$(cat /tmp/drift-pr-body.md)"
   

PR body format

## Upstream config drift detected — <vendor>

Net-new examples found in upstream docs that pass `testagent <vendor> validate --strict`. Adding to the corpus.

### Added
- `testdata/upstream-examples/<vendor>/<slug>.json` (from <source-url>)

### Drifted (existing fixture diverges from upstream)
- `testdata/upstream-examples/<vendor>/<existing>.json` — <description of change>

### Skipped — fail --strict
- N examples with <field> field (not in testagent's allowlist); tracked separately in `update-compatibility`.

🤖 Generated by `.github/workflows/upstream-drift.yml`

Hard constraints

• Do NOT vendor a block that fails --strict. Surface it as an update-compatibility follow-up.
• Do NOT modify existing fixtures. Only add new files or replace drifted ones.
• Do NOT open a PR if one already exists for this vendor (de-dupe check in step 6).
• Do NOT post comments or external messages; this skill produces a draft PR only.

File layout

.claude/skills/verify-upstream-config-examples/
├── SKILL.md        # this file
└── sources.md      # per-vendor source URLs and extraction notes

The shell-script implementation lives at scripts/upstream-drift.sh (invoked by .github/workflows/upstream-drift.yml). The skill and the script must stay in sync.