afdocs-fix

# AFDocs Fix Apply automated fixes for issues found by the `afdocs-audit` skill. This skill reads an AFDocs audit report and remediates fixable issues. ## Agent-doc quality contract The standing PR this skill maintains follows the shared v1 agent-doc quality contract in `.agents/references/doc-quality-policy.md`: apply the `warpy-factory` label and add the `## Documentation risk` block (`.agents/skills/doc_quality_policy/finalize_pr_contract.py build`). These fixes touch site infrastructure (middleware, integrations, config), not content claims, and are typically `engineering-review-required` unless the change is provably a mechanical config update. ## Prerequisites 1. Run the audit skill first to produce a report: ```bash node .agents/skills/afdocs-audit/scripts/afdocs_audit.mjs --output /tmp/afdocs-report.json ``` 2. Read the report and cross-reference against `afdocs-audit/references/known-exceptions.md` to identify which issues are genuine (vs. allowlisted). ## Fix procedures by check For each failing or warning check, follow the procedure below. Skip checks listed in `known-exceptions.md`. ### llms-txt-directive-html (Content Discoverability) **What's wrong**: No agent-facing directive pointing to llms.txt in the HTML pages. **Fix**: Add a visually-hidden element in `src/components/CustomHeader.astro` before the `.header` div: ```html <div class="llms-directive sr-only" aria-hidden="true"> For the complete documentation in markdown, see <a href="/llms.txt">llms.txt</a>. Markdown versions of each page are available by appending .md to any URL. </div> ``` The `.sr-only` class is Starlight's built-in screen-reader-only utility. The `sanitizeRoot` function in `docs-markdown-integration.js` strips `.sr-only` elements during HTML → markdown conversion, so this directive will appear only in the HTML version (the markdown version gets its own directive — see below). **Files**: `src/components/CustomHeader.astro` ### llms-txt-directive-md (Content Discoverability) **What's wrong**: No llms.txt directive in the generated markdown pages. **Fix**: In `src/integrations/docs-markdown-integration.js`, in the `convertHtmlToMarkdown` function, prepend a blockquote before the title: ```javascript const llmsDirective = '> For the complete documentation index, see [llms.txt](/llms.txt).\n' + '> Markdown versions of each page are available by appending .md to any URL.'; const sections = [llmsDirective, `# ${normalizeWhitespace(title)}`]; ``` **Files**: `src/integrations/docs-markdown-integration.js` ### content-negotiation (Markdown Availability) **What's wrong**: Server ignores `Accept: text/markdown` header and returns HTML. **Fix**: Create Astro middleware at `src/middleware.ts` that uses the existing `shouldServeMarkdown()` helper from `src/lib/docs-markdown.js`: ```typescript import { defineMiddleware } from 'astro:middleware'; import { shouldServeMarkdown, isEligibleDocHtmlPath, getMarkdownPathFromHtmlPath, } from './lib/docs-markdown.js'; export const onRequest = defineMiddleware(async (context, next) => { const { request, url } = context; if (!isEligibleDocHtmlPath(url.pathname)) return next(); if (!shouldServeMarkdown(request)) return next(); const mdPath = getMarkdownPathFromHtmlPath(url.pathname); return context.rewrite(mdPath); }); ``` The `shouldServeMarkdown` helper checks both the `Accept` header (for `text/markdown` and `text/plain`) and user-agent tokens (ChatGPT, Claude, Cursor, etc.). **Files**: `src/middleware.ts` (new file) ### llms-txt-coverage (Observability) **What's wrong**: llms.txt covers less than 95% of sitemap pages. **Fix**: Update the `customSets` paths in `astro.config.mjs` under `starlightLlmsTxt()`: 1. Check that every content directory under `src/content/docs/` has a matching customSet entry. 2. Common mismatches: - Directory was renamed but customSet path wasn't updated (e.g., `university/**` → `guides/**`) - New content directories added without a corresponding customSet - Sub-paths excluded too aggressively (e.g., `support-and-community/community/` pages) 3. For large pages that cause `hast-util-to-text` stack overflows: the `exclude` option only applies to `llms-small.txt`, NOT `llms-full.txt`. Do not add pages to `exclude` expecting them to be skipped in `llms-full.txt`. **Diagnostic steps**: ```bash # List content directories ls src/content/docs/ # Compare against customSets in astro.config.mjs grep -A1 "label:" astro.config.mjs | grep "paths:" ``` **Files**: `astro.config.mjs` ### markdown-url-support (Markdown Availability) **What's wrong**: Some pages don't return markdown when `.md` is appended to the URL. **Fix**: Check that `src/integrations/docs-markdown-integration.js` and `src/pages/[...slug].md.ts` are both present and correctly configured. The integration generates static `.md` files at build time; the page route serves them in dev mode. **Files**: `src/integrations/docs-markdown-integration.js`, `src/pages/[...slug].md.ts` ### markdown-link-portability (Content Structure) **What's wrong**: Generated markdown contains root-relative or path-relative links, or `.md` links resolve to HTML/error content. **Fix**: Update `src/integrations/docs-markdown-integration.js` to rewrite internal links to absolute `https://docs.warp.dev/...` URLs after Turndown conversion. Preserve same-document fragment links (`#section`) unchanged. Add a test to `src/integrations/docs-markdown-integration.test.js` that verifies an internal link is absolute, then build and fetch a representative `.md` page to verify `Content-Type: text/markdown`. **Files**: `src/integrations/docs-markdown-integration.js`, `src/integrations/docs-markdown-integration.test.js` ### single-fetch-completeness (Page Size and Truncation Risk) **What's wrong**: A served markdown response is paginated without a working, absolute continuation declared near the top of the content. **Fix**: Do not copy HTML-UI pagination into markdown when the complete resource fits within the page-size limits. Otherwise add an absolute continuation link before the main content and test that it returns substantive markdown rather than an error page. Audit the markdown generator and any data-driven route that produces paginated output. **Files**: `src/integrations/docs-markdown-integration.js`, `src/pages/[...slug].md.ts`, affected data-driven page or endpoint ### page-size-transfer / embedded-data-serialization (Page Size and Content Structure) **What's wrong**: HTML responses are large before conversion, or static bulk data (tables, JSON, base64) dominates content that agents receive. **Fix**: Treat this as a design or rendering-pipeline change, not an automatic content split. Attribute the bytes to inline framework payloads, generated tables, or data blocks first. Move large payloads behind on-demand requests, split generated tables into self-contained topic pages, and keep prose before bulk content. Do not replace a complete markdown page with pagination windows. **Files**: Varies by the reported page and its rendering/data source. ### bot-protection-interference (Authentication and Access) **What's wrong**: Sustained automated requests receive challenge pages, time out, or are blocked after volume thresholds are reached. **Fix**: This requires CDN/WAF configuration, not a docs-repository code change. Exempt public docs paths from behavioral enforcement or scope enforcement to interactive product surfaces. Prefer explicit `429` responses with `Retry-After` over challenge interstitials or stalled responses. A partial scan must not be compared with prior audit scores. **Files**: Vercel Firewall or the active CDN/WAF configuration; see `afdocs-audit/references/vercel-firewall-challenge.md`. ### http-status-codes (URL Stability) **What's wrong**: The site returns 200 for non-existent pages (soft 404). **Fix**: Ensure `public/404.html` or an Astro 404 page exists and returns a proper 404 status code. In Vercel, configure `cleanUrls` and ensure the adapter handles 404 responses correctly. **Files**: `vercel.json`, `src/pages/404.astro` (if it exists) ### MCP Server Discoverable **What's wrong**: No MCP server discovery endpoint found. **Fix**: Add a static discovery file at `public/.well-known/mcp.json`: ```json { "name": "Warp Documentation", "description": "Search and retrieve Warp documentation.", "url": "https://warp.mcp.kapa.ai" } ``` This points to the existing Kapa-hosted MCP server (OAuth-protected). No custom implementation needed. **Files**: `public/.well-known/mcp.json` (new file) ## Checks with no automated fix These checks require infrastructure or design changes that can't be automated: - **content-start-position** — Inherent to Starlight's layout. Mitigated by content negotiation and llms.txt directives. See `known-exceptions.md`. - **page-size-markdown / page-size-html** — Requires editorial decision to split long pages. Flag in the report but do not auto-fix. - **page-size-transfer / embedded-data-serialization** — Requires rendering-pipeline or information-architecture decisions after attributing the bulk bytes. - **bot-protection-interference** — Requires CDN/WAF configuration and turns the scan into a partial observation rather than a deployable docs change. - **section-header-quality** — Content-level change requiring human judgment. ## Applying fixes This skill maintains **one** long-lived fixes PR rather than one per run — see "One standing PR per automation" in `.agents/references/skill-authoring-guidelines.md`. 1. Look for an existing open PR before creating a branch: ```bash gh pr list --repo warpdotdev/docs --state open \ --search 'AFDocs fixes in:title' --json number,headRefName ``` 2. Check out the standing branch. If the PR exists, continue on its branch and rebase; otherwise create it from `main`: ```bash git fetch origin git checkout afdocs-fixes 2>/dev/null || git checkout -b afdocs-fixes origin/main git rebase origin/main ``` 3. Apply the fixes for each failing check (skip allowlisted checks). If a fix on the existing branch already addresses a check that is still failing, do not duplicate it — the audit may have run before the PR merged. 4. Validate: `npm run build` (the build must succeed). 5. Commit with the prefix: `AFDocs fixes: <summary of what was fixed>` 6. Push. If the PR already exists the push updates it; otherwise open one with `gh pr create`. Never leave two open AFDocs PRs. If you find more than one, consolidate onto `afdocs-fixes` and close the extras with a comment pointing at the survivor. ## PR conventions - Title must be prefixed with `AFDocs fixes:` and must not contain a date — a dated title defeats the title search in step 1 and produces a new PR every run - Include the audit score (before/after if known) in the PR description - When updating an existing PR, append the new run's score and fixes under the existing headings rather than adding duplicate headings, which `check_pr_body.py` rejects - Include the co-author line: `Co-Authored-By: Oz <oz-agent@warp.dev>`