validate-ui-refs

# Validate UI References This skill scans Warp's Astro Starlight documentation for references to UI paths (e.g. `Settings > AI > Active AI`) and Command Palette command names (e.g. "Open Theme Picker"), then validates them against a snapshot of known-valid paths extracted from the public warp client repo ([warpdotdev/warp](https://github.com/warpdotdev/warp)). ## Running the Check From the docs repo root: ```bash python3 .agents/skills/validate_ui_refs/validate_ui_refs.py --all ``` ### Options - `--check-paths`: Only validate UI menu paths (Settings, File, View, Warp Drive) - `--check-commands`: Only validate Command Palette names - `--check-format`: Check that UI paths use the canonical bold format: **Settings** > **AI** > **Active AI** - `--all`: Run all checks (default) - `--fix`: Auto-fix high-confidence issues (e.g. case mismatches) - `--create-pr`: Create a branch and PR with auto-fixes (requires `gh` CLI) - `--slack-notify`: Post results to `#growth-docs` Slack channel when unfixed issues remain (requires `SLACK_BOT_TOKEN` env var; channel is hardcoded in the script) - `--slack-channel ID`: Override the default Slack channel (`C09BVK0PL3Y`) - `--self-test`: Run internal sanity checks against the current snapshot and exit (no warp checkout needed) - `--include-changelog`: Include `changelog/` in the scan (excluded by default since it's a historical record) - `--refresh-valid-paths`: Re-extract valid paths from the warp client repo and update `valid_paths.json` - `--warp PATH`: Path to the public warp client repo (auto-detected as a sibling of the docs repo named `warp`, with `warp-internal` as a fallback, or the `WARP_REPO_PATH` env var). `--warp-internal-path` is a deprecated alias, and `WARP_INTERNAL_PATH` is still honored as a deprecated env var fallback. - `--output FILE`: Save results to a JSON file ### Quick path-only check: ```bash python3 .agents/skills/validate_ui_refs/validate_ui_refs.py --check-paths ``` ### Full check with auto-fix and PR: ```bash python3 .agents/skills/validate_ui_refs/validate_ui_refs.py --all --fix --create-pr ``` ## Output Format The script outputs a report like: ``` === UI REFERENCE VALIDATION REPORT === Generated: 2026-02-19T00:17:00Z Files scanned: 174 ### SETTINGS PATH ISSUES (5 found) ❌ "Current Theme" is not a known sub-section of Appearance themes.md:26 Path: Settings > Appearance > Current Theme Suggestion: Valid sub-sections: Themes, Icon, Window, Input, Panes, Blocks, Text, Cursor, Tabs, Full-screen Apps ⚠️ "active ai" is not a known sub-section of AI codebase-context.md:34 Path: Settings > AI > active ai Suggestion: Did you mean "Active AI"? (score: 0.92) ### COMMAND PALETTE ISSUES (1 found) ❌ UNMATCHED COMMAND themes.md:26 Reference: "Open Theme Creator" Did you mean "Open theme picker"? (score: 0.85) ``` ## Refreshing Valid Paths The `valid_paths.json` file is a static snapshot of valid UI paths. To update it from the latest warp client source: ```bash python3 .agents/skills/validate_ui_refs/validate_ui_refs.py --refresh-valid-paths --warp /path/to/warp ``` This parses: - `SettingsSection` enum and `Display` impl from `settings_view/mod.rs` - `Category::new(...)` and `build_sub_header(...)` calls from settings page files - `render_body_item::<...>(...)` and `render_dropdown_item(...)` call sites in each page's own (non-shared) source file, to build a control/widget-label-to-page map (see "Control relocation detection" below) - `EditableBinding::new(...)` registrations and the enable/disable actions generated by `ToggleSettingActionPair::new(...)` The macOS menu bar and Warp Drive sections are maintained as manual lists in `valid_paths.json` since they change infrequently. ### Control relocation detection Validation used to stop at the Settings navigation hierarchy (section/umbrella/subpage + `Category`/`build_sub_header` sub-sections) and treat any trailing path segment as a free-form toggle/setting name, allowing it unconditionally. That let a stale path survive a control relocation indefinitely whenever the old section itself still exists: `Settings > Features > General > Choose an editor to open file links` kept validating after that control moved to `Settings > Code > Editor and Code Review`, because `Features > General` never stopped being a real section (see [QUALITY-2052](https://linear.app/warpdotdev/issue/QUALITY-2052)). `valid_paths.json`'s `settings_sections` entries now also carry a `controls` list — the exact widget/dropdown label strings each page renders, extracted from `render_body_item::<...>(...)` and `render_dropdown_item(...)` call sites (resolving simple `const NAME: &str = "...";` indirection). `validate_ui_path` checks any unrecognized trailing segment against this control-to-page map: when it names a control now owned by a different page than the one documented, the path fails with a suggestion pointing at the control's actual page. An unrecognized segment that matches no known control is still allowed as a free-form toggle name, and a section with no matching control in the trailing position (e.g. `Settings > Features > General` alone) stays valid. Controls are only extracted from a page's own, unshared source file — a file backing multiple pages (`ai_page.rs`, `mcp_servers_page.rs`, `platform_page.rs`) is skipped for control extraction, the same constraint noted above for sub-section accuracy, so a shared file's labels are never misattributed to every page it backs. This is the one place UI-path relocation detection lives; `missing_docs`'s settings audit only checks `toml_path` key coverage and defers here for navigation-path accuracy (see that skill's "Adjacent checks" section). ### Sub-section accuracy for shared-source-file pages The extractor assigns sub-sections by reading `Category::new(...)` calls from each page's source file. When multiple Settings subpages share the same source file (e.g. `ai_page.rs` is used by Warp Agent, Profiles, Knowledge, and Third party CLI agents), all category calls in that file are assigned to all of those subpages — producing incorrect sub-section lists. After running `--refresh-valid-paths`, manually verify the sub-sections for these pages against the live Warp Settings UI, or invoke the `verify-settings-subsections` skill (requires computer use) which automates this check. See `.agents/skills/verify-settings-subsections/SKILL.md`. ## What Gets Validated ### UI Paths - **Settings paths**: `Settings > [Section] > [Sub-section]` — validates section and sub-section names against the `SettingsSection` enum - **macOS menu bar**: `File > ...`, `View > ...`, `Warp > ...` — validates against known menu items - **Warp Drive**: `Warp Drive > ...`, `Personal > ...` — validates against known spaces and object types - **Multiple markdown formats**: backtick-wrapped, bold, italic, and bare inline paths ### Format Consistency - All UI paths should use per-segment bold formatting: **Settings** > **AI** > **Active AI** - Backtick formatting (`` `Settings > AI` ``), full bold wrapping (`**Settings > AI**`), italic, and bare formats are flagged - Auto-fix with `--fix` converts non-canonical paths to the correct bold format ### UI Element Formatting - Clickable UI elements (buttons, toggles, links, dropdowns) should be bold, not backtick: Click **Save**, not Click `Save` - Detected via action keywords: click, select, toggle, enable, disable, choose, check, uncheck, expand, collapse, open, close, tap - Handles prepositions: "click on `X`", "click the `X`" are also caught - Keyboard keys stay in backticks (press `Enter`, hit `Esc`) — "press" and "hit" are excluded from the check - Code-like content (CLI flags, paths, CamelCase identifiers, keyboard shortcuts) is automatically excluded - Auto-fix with `--fix` converts backtick UI elements to bold ### Command Palette Names - Quoted strings near "Command Palette" mentions - `Command Palette > CommandName` arrow patterns - Fuzzy matching with suggestions for near-misses ### Exclusions - `changelog/` directory (historical record; use `--include-changelog` to opt in) - `_book/` and `node_modules/` build artifacts - External product paths (GitHub, Slack, etc.) are filtered by context ## Auto-Fix When run with `--fix`, the script automatically corrects: - **Case mismatches**: e.g. `Settings > ai` → `Settings > AI` - **Non-canonical UI path formats**: backtick, italic, bare → per-segment bold - **Backtick UI elements**: e.g. Click `Save` → Click **Save** - Only path case fixes with confidence ≥ 0.9 are applied; format fixes are always applied Fixes that require manual review (e.g. renamed sections, removed features) are reported but not auto-fixed. ## Slack Notifications Slack notifications are designed for automated runs, not ad-hoc usage. When running the skill manually, review results directly in the terminal output. The `--slack-notify` flag posts a summary to `#growth-docs` when unfixed issues remain after a run **and** that set of issues differs from what was already reported (tracked via an open `fix/ui-refs-*` PR's body, or `last_notified_signature` in `valid_paths.json` once such a PR merges). If the scan is clean (0 issues), or the exact same unresolved issues were already reported, no notification is sent — this keeps the weekly reconciliation schedule from re-posting an identical report while a fix PR awaits review. ### Setup For **automated runs**, the Slack token is provided by the Docs Agent Oz environment via `BUZZ_SLACK_TOKEN` — no GHA secret needed. For **manual runs** from the command line, export a Slack bot token with `chat:write` scope before running: ```bash export SLACK_BOT_TOKEN=xoxb-... python3 .agents/skills/validate_ui_refs/validate_ui_refs.py --all --slack-notify ``` ## Automated refresh `valid_paths.json` is refreshed automatically via the `refresh-ui-paths` GitHub Actions workflow (`.github/workflows/refresh-ui-paths.yml`). ### How it works The workflow runs on three triggers, only one of which is throttled: 1. A push to `master` in `warpdotdev/warp` that touches `app/src/settings_view/**` sends a `repository_dispatch` event (`settings-ui-changed`) to `warpdotdev/docs`. This runs immediately, with no schedule-based delay. 2. A weekly safety-net schedule (Fridays at `15:15 UTC`) reconciles the snapshot's recorded `source_sha` against the current `warpdotdev/warp` `master` HEAD, catching a missed or failed `repository_dispatch` within seven days. It refreshes only when the recorded and current SHAs differ; a no-change week exits without dispatching an agent. 3. `workflow_dispatch` triggers immediately, the same as a source dispatch (see "Manual trigger" below). When a refresh runs (from any trigger): 4. The `refresh-ui-paths` GHA workflow dispatches an Oz cloud agent to the Docs Agent environment (`K5KStCm5aYvhfBJb8cHol6`). 5. The cloud agent runs `--refresh-valid-paths` using the `warpdotdev/warp` checkout available in that environment. 6. If the snapshot changed, the agent runs `--all --fix --create-pr --slack-notify` to validate and auto-fix. `--create-pr` commits and opens/updates a PR whenever there's a fix, a refreshed snapshot (e.g. a `source_sha` bump), or a newly-changed set of unresolved issues to report — not only when there's an auto-fixable doc issue, so the recorded `source_sha` keeps advancing even on a quiet run. `--slack-notify` only posts to `#growth-docs` when the unresolved-issue set is new (see "Slack Notifications" above). 7. If the snapshot is unchanged, the agent exits with no-op. ### Secrets required | Secret | Where | Status | Notes | |---|---|---|---| | `DOCS_REPOSITORY_DISPATCH_TOKEN` | `warpdotdev/warp` GHA secrets | ✅ Already provisioned | Sends the `settings-ui-changed` event to docs via `peter-evans/repository-dispatch` | | `WARP_API_KEY` | `warpdotdev/docs` GHA secrets | ✅ Already provisioned | Used by the GHA workflow to dispatch the Oz cloud agent | | `BUZZ_SLACK_TOKEN` | Docs Agent Oz environment | ✅ Already provisioned | Used by the cloud agent for `#growth-docs` Slack notifications (same bot that posts 404 reports, etc.) | `warpdotdev/warp` (the OSS repo) is used for snapshot extraction since that's also where the trigger fires from. The Docs Agent Oz environment has `warpdotdev/warp` in its configured repos, or the agent will clone it if not present. No private repo access needed. ### Manual trigger To trigger the workflow manually (e.g., if the PAT expired or a migration was missed): 1. In the `warpdotdev/docs` repo, go to **Actions** > **Refresh UI paths snapshot** > **Run workflow**. 2. Or run locally: ```bash python3 .agents/skills/validate_ui_refs/validate_ui_refs.py \ --refresh-valid-paths \ --warp /path/to/warp ``` ## Agent-doc quality contract The `--create-pr` auto-fix path is a direct PR-creation code path (`create_pr()` in `validate_ui_refs.py`), so it stamps the `warpy-factory` label and the `## Documentation risk` block itself — always `low` risk, since an auto-fix only ever corrects UI-reference casing/formatting to an already-canonical name, never product meaning. See `.agents/references/doc-quality-policy.md`. ## Dependencies - Python 3.7+ - `requests` (for Slack notifications): `pip3 install requests` - `gh` CLI (for PR creation) - A checkout of the public warp client repo ([warpdotdev/warp](https://github.com/warpdotdev/warp)) (only for `--refresh-valid-paths`)