check-docs-drift

name	check-docs-drift
description	Detect upstream changes to polkadot-docs tutorials, classify drifts, and sync dependency versions from upstream variables.yml.

Check Docs Drift

Scan all polkadot-docs/ test harnesses, compare their pinned docs_commit against the latest upstream commit, classify any drifts, and sync dependency versions from the upstream variables.yml.

---

Metrics Tracking

Throughout the entire run, maintain running counters to produce a Run Report at the end:

• Wall-clock time: Record date +%s at the start and end of each phase to measure duration.
• Counts per phase: READMEs scanned, drifts found, subagents spawned, diff lines fetched, gh api calls made, files read/modified, npm install runs, versions bumped.

Track these as you go — do not retroactively estimate. Report them in Phase 8.

---

Phase 1: Scan & Detect Drift

1. Find all polkadot-docs/**/README.md files. Exclude node_modules, .test-workspace, and the top-level polkadot-docs/README.md.

2. For each README, parse the YAML frontmatter and extract:

   • source_github or source_repo (both field names are used — check both)
   • docs_commit
   • title

3. Categorize each README:

   • Trackable: has a source URL (source_github or source_repo) AND docs_commit
   • Missing pin: has a source URL but no docs_commit
   • Untracked: no source URL — skip silently (these are cloned template READMEs, etc.)

4. For each trackable README, extract the file path from the GitHub URL (strip everything up to and including /blob/master/) and fetch the latest commit SHA:

   gh api "repos/polkadot-developers/polkadot-docs/commits?path={FILE_PATH}&per_page=1&sha=master" --jq '.[0].sha'
   

5. Compare the fetched SHA against docs_commit. Build a list of drifted entries (old SHA, new SHA, file path, title, README path).

---

Phase 2: Analyze Drifts

Optimization: Before spawning subagents, group drifted files by their (old_sha, new_sha) pair. Files sharing the same commit range can be analyzed in a single subagent with one API call, avoiding redundant fetches. This is common when upstream makes a bulk change (e.g., badge updates) that touches many files at once.

For each drift (or group of drifts sharing a commit range), spawn a parallel subagent (via the Agent tool) that:

1. Fetches the patch:

   gh api "repos/polkadot-developers/polkadot-docs/compare/{old_sha}...{new_sha}" --jq '.files[] | select(.filename == "{FILE_PATH}") | .patch'
   

   If the patch is empty or truncated, fetch the raw file at both SHAs and diff locally.

2. Classifies the diff as COSMETIC or SUBSTANTIVE:

   COSMETIC (safe to auto-bump docs_commit):

   • Whitespace, indentation, blank lines
   • Markdown formatting changes (bold/italic syntax, heading levels without content change)
   • Typo corrections in prose (NOT in code blocks)
   • Link URL updates where destination content is unchanged
   • Badge/shield URL updates
   • Comment rewording with no semantic change
   • Content reordering without additions/removals

   SUBSTANTIVE (requires test harness review):

   • Any change inside fenced code blocks (commands, code, config)
   • New or removed numbered steps
   • Changed dependency names or versions
   • Changed CLI commands, flags, or arguments
   • New prerequisites or tool requirements
   • Changed file paths or directory structures
   • New or removed sections
   • Changed expected output or behavior descriptions

3. Returns: verdict (cosmetic/substantive), one-line summary, list of key changes.

---

Phase 3: Present Results

Display a summary table grouped into sections:

1. Substantive Drifts (action needed)

Guide	Drift	Summary
{title}	view diff	{one-line summary}

2. Cosmetic Drifts (safe to bump)

Guide	Drift	Summary
{title}	view diff	{one-line summary}

3. Up to Date

List titles of guides where docs_commit matches the latest SHA.

4. Missing docs_commit

List guides that have a source URL but no docs_commit field — suggest adding it.

---

Phase 4: Auto-Bump & Report

Act autonomously — do not prompt the user for confirmation:

• Cosmetic drifts: Automatically update docs_commit in each affected README to the latest SHA, commit, and push.
• Substantive drifts: Do NOT auto-bump. Report them in the results and the GitHub issue for manual review.
• Missing docs_commit: Report them in the results. Do not add the field automatically — this is an informational note for the user.

---

Phase 5: Create GitHub Issue

After presenting results and performing any user-approved actions, create a GitHub issue summarizing the findings so a developer can resolve the substantive drifts later.

• Title: [Docs Drift] {N} tutorial(s) updated upstream
• Label: docs-drift
• Check for an existing open issue with the docs-drift label first. If one exists, comment on it instead of creating a new one.
• Body should include:
  • The full results table from Phase 3 (substantive drifts, cosmetic drifts, up to date, missing pins)
  • For substantive drifts: diff links and summaries of what changed
  • For cosmetic drifts: note whether they were auto-bumped or still pending
  • A call to action: "Review the substantive drifts and update test harnesses as needed."

# Check for existing issue
gh issue list --state open --label "docs-drift" --json number --jq '.[0].number // empty'

# Create or comment
gh issue create --title "{title}" --body "{body}" --label "docs-drift"
# OR
gh issue comment {number} --body "{body}"

---

Phase 6: Sync Dependency Versions

Compare our versions.yml against the upstream polkadot-docs variables.yml.

1. Fetch the upstream file:

   gh api repos/polkadot-developers/polkadot-docs/contents/variables.yml --jq '.content' | base64 -d
   

2. For each entry in our versions.yml, find the matching entry in upstream variables.yml and compare versions. The structures differ — map between them:

   • polkadot_sdk.release_tag ↔ dependencies.repositories.polkadot_sdk.version
   • parachain_template.crates.* ↔ dependencies.repositories.polkadot_sdk_parachain_template.subdependencies.* and dependencies.crates.*
   • zombienet.version ↔ dependencies.repositories.zombienet.version
   • crates.* ↔ dependencies.crates.*
   • javascript_packages.* ↔ dependencies.javascript_packages.*

3. Build a list of version mismatches (our version vs upstream version).

4. For each outdated entry:

   • Update versions.yml
   • Find all package.json files (in polkadot-docs/, recipes/, migration/, and dot/sdk/templates/) that reference the same package at the old version and update them
   • Run npm install --package-lock-only in each modified harness directory to regenerate lockfiles

5. If any entries exist in upstream but not in our versions.yml, check whether that package is used in any harness package.json. If so, add it to versions.yml and report it.

6. Present a summary table of version changes made, and any new entries added.

---

Phase 7: Self-Improvement

After completing the pipeline, reflect on what happened during this run:

1. Were there API or parsing issues? Did any README use a frontmatter field name not covered by the skill? Did the GitHub API return unexpected results?
2. Was the batching effective? Did grouping by commit range save API calls, or were there edge cases?
3. Were the classifications accurate? Did any diff get misclassified as cosmetic when it was substantive, or vice versa?
4. Were there missing categories? Did the rubric fail to cover a type of change encountered in the diffs?

If you identified concrete improvements, create a draft PR on a separate branch (chore/improve-check-docs-drift-skill) with changes to this skill file.

When writing improvements, follow the Claude Code skills documentation and these best practices:

• Keep SKILL.md directive, not prescriptive — say "study this reference file and adapt" instead of embedding full code templates.
• Keep the skill concise (under ~200 lines).
• Reference existing implementations rather than duplicating patterns.
• Keep the skill autonomous — never add steps that prompt or wait for user input.
• Always create draft PRs (gh pr create --draft) — never create ready-to-merge PRs from skill runs.

Include in the PR description:

• What triggered the improvement (the specific failure or gap encountered)
• What changed and why

---

Phase 8: Run Report

Display a final metrics summary so the user can see where time and tokens were spent:

## Run Report

| Phase | Duration | Key stats |
|-------|----------|-----------|
| 1. Scan & Detect | 12s | 15 READMEs scanned, 4 drifted |
| 2. Analyze Drifts | 38s | 3 subagents, 2 batches, 847 diff lines |
| 3. Present Results | 2s | — |
| 4. Auto-Bump | 5s | 2 cosmetic bumps committed |
| 5. GitHub Issue | 3s | Created #248 |
| 6. Version Sync | 22s | 4 bumped, 10 package.json, 8 lockfiles |
| 7. Self-Improvement | 8s | No improvements identified |
| **Total** | **90s** | **3 subagents, 847 diff lines, 15 API calls** |

### Bottleneck notes
- Phase 2 consumed most time due to 1 large diff (623 lines)

Guidelines:

• Always populate with real numbers from the run — never estimate or omit.
• In Bottleneck notes, call out the single largest time/token consumer and suggest whether it could be optimized (e.g., better batching, skipping unchanged sections, splitting large diffs).
• If any phase took 0s or had no work, show it as — to confirm it was reached.