auditkit

name	auditkit
description	Comprehensive, evidence-based website audit that actually fetches and parses the live site instead of guessing from memory. Use this whenever the user wants to audit, review, grade, or improve a website — for SEO, technical health, Core Web Vitals / performance, accessibility (WCAG), conversion rate (CRO), security headers, structured data, or AI-search readiness (GEO / AEO). Triggers on requests like "audit my site", "run an SEO audit", "check my site's accessibility", "why is my site slow", "how does my site look to AI crawlers", "score my website", or dropping in a URL and asking what's wrong with it. Produces a weighted 0–100 score, a prioritized fix list with copy-paste code, and a multi-audience report. For sites the user owns or is authorized to test.
license	Apache-2.0
compatibility	Best with a code-execution environment (Bash + Python 3) so the bundled scripts can run. Python deps: requests, beautifulsoup4. Optional power-ups: a free Google Cloud API key (unlocks PageSpeed Insights + CrUX real-user data), and a headless browser for JS-rendering checks. Degrades gracefully when any of these are missing.

AuditKit — Website Audit

AuditKit runs a real audit: it fetches robots.txt, sitemaps, HTML, headers, and JSON-LD from the live site, runs deterministic scripts to measure things, compares them against hard-coded thresholds, and produces a weighted, prioritized, decision-ready report. The whole thing runs free — no paid SEO suite required. It is honest about what genuinely can't be measured for free (backlinks, keyword rankings, real-user field data without a key) instead of faking it.

Read this whole file first, then work the phases in order. Load reference files only when a phase tells you to — they cost no context until read. Run the bundled scripts rather than re-deriving their logic; they exist so every audit doesn't reinvent the wheel.

Core principles (these are what make the output credible)

• Measure, don't guess. Every finding must trace to something you actually fetched or computed. "Your LCP is 3.8s (measured)" beats "your site may be slow." If you couldn't measure something, say so — never present an absent or failed check as a pass.
• Never let an estimate masquerade as a measurement. When a category is scored from lab simulation instead of real-user data, or from an automated scan instead of full manual review, state that confidence level right next to the sub-score.
• The score is the product, but the fix list is why they came. Lead with the number, then give a prioritized action plan sorted by points-recoverable-per-effort. For every point lost, generate one concrete fix — and where the fix is mechanical, output the literal artifact (the rewritten tag, the JSON-LD block, the corrected header) rather than prose.
• Be evidence-based, not hype-based. This is the differentiator against generic "AI SEO" tools. Report llms.txt honestly (useful for dev/agent-facing sites, not a proven ranking or citation lever). Put the real GEO weight on robots.txt AI-bot correctness, direct-answer content structure, and schema/entity consistency.
• Degrade gracefully. If an external API is down or a key is missing, fall back, note the limitation, and keep going. Endpoints in this space die regularly — treat every external call as potentially unavailable.
• Narrate as you go. This audit takes a while and makes many network calls, so don't run silently. Before each phase, say in one line what you're about to do and why; after it, report what you found in a sentence ("Crawled 30 of ~610 pages — sitemap is valid and fresh; 0 broken links so far"). Use the scripts' --progress output for long crawls. The user should always know which phase you're in and never wonder whether it's stuck. Keep these to short status lines — the full detail goes in the final report, not the running commentary.

Phase 0 — Intake, consent & scoping

Before fetching anything beyond a couple of pages, settle these. Tell the user you're setting up, then:

1. Check the environment. Run scripts/check_env.py first. It reports which capabilities are live (core crawl/parse) and which will degrade (no CrUX key → lab-estimate performance; no Node/axe-core → alt-text-only accessibility). Tell the user in one line what's active and what would unlock the rest — this is what lets the report honestly label estimated vs. measured categories later. If core deps are missing, point them at the install guide (INSTALL.md) before continuing.
2. Pick the mode (this is the first thing to settle — it changes everything downstream):
   • Single-page — audit exactly one URL. Right for a spot-check, a single landing/campaign page, or a quick "what's wrong with this page." Run scripts/crawl.py <url> --single. Fast, no crawl. Phases still run, just scoped to the one page; skip site-wide items (sitemap coverage, orphan pages, cross-page duplication) and say so.
   • Full-site discovery — discover and sample across the whole site. Right for "audit my site." Run scripts/crawl.py <url> --budget <N> --progress. It seeds from the sitemap, stratifies the sample across sections, and reports coverage. Infer the likely mode from how the user asked ("audit this page" → single; "audit my site" → full-site) and confirm it, rather than defaulting silently.
3. Capture inputs. The target URL (required). Site type (e-commerce / SaaS / blog / local / portfolio / marketplace) — infer it and confirm. Optional power-ups: a Google Cloud API key (PSI + CrUX), and any GSC / Ahrefs / SEMrush / analytics CSV exports the user can paste (you can't fetch that data, but you're an excellent analyst of it if provided). Up to 3 competitor URLs for comparison mode.
4. Ethical-use gate (required, not optional). Ask the user to confirm they own the site or have explicit permission to audit it. You cannot verify this in a chat context, so an affirmative confirmation is the mechanism — the same trust model nmap/nikto/ZAP rely on. If they can't confirm, run only the lightest non-invasive checks (a single-page fetch of public metadata) and skip the wider crawl and the security self-scan entirely.
5. Set full-site scope (skip if single-page). Default budget: 50 pages unless the user raises it. The crawler already selects deliberately — homepage first, then a sitemap-stratified sample across sections rather than discovery order. Always report the sampling method and coverage ("audited 47 of ~4,200 pages, sampled across all 6 sections"); the crawler emits a coverage_note for exactly this — surface it, don't present a sample as a full audit.
6. Politeness is built in. The scripts identify with a descriptive User-Agent, respect any Crawl-delay, cap concurrency, and hard-stop at the budget. On a 403/429, back off and report "the site's bot protection blocked this check" rather than retrying aggressively.

Read references/responsible-use.md for the full rules of engagement (no auth bypass, no paywalls/CAPTCHAs, copyright handling, the separate security-scan consent gate).

Phase 1 — Crawl & Technical SEO

Run the crawler in the mode chosen in Phase 0 — scripts/crawl.py <url> --single for one page, or scripts/crawl.py <url> --budget <N> --progress for full-site. Add --out crawl.json so the other scripts (check_links.py --crawl, validate_jsonld.py --crawl) can reuse it. It fetches robots.txt, the sitemap, and the sampled pages, recording per page: status codes, redirect chains, canonical / hreflang / meta-robots tags, indexability, title/description/headings, images, internal + external links, and JSON-LD blocks. It respects robots.txt and politeness limits. Output is JSON you then analyze. Tell the user how many pages came back and the coverage before moving on.

Then check:

• Indexability (highest leverage): robots.txt Disallow rules, noindex, canonical correctness, www vs non-www and HTTP vs HTTPS duplication.
• AI-crawler access in robots.txt — see Phase 6; the raw directives come from this crawl.
• Sitemap: valid XML, indexable-only URLs, <lastmod> freshness, URL count vs pages found.
• Redirects & status: flag chains > 1 hop and any 4xx/5xx.
• Broken links — run scripts/check_links.py on the collected hrefs. Report as three separate findings: broken internal links, broken external links, broken images. De-duplicate the same broken URL into one finding with a "linked from" list.
• Structured data: run scripts/validate_jsonld.py on each page's JSON-LD; validate required fields against references/schema-templates.md. No free Rich Results API exists — validate locally, then link the user to Google's UI test for confirmation.
• JS-rendering / SPA check: compare the raw HTML response against the rendered DOM. AI crawlers (GPTBot, ClaudeBot, OAI-SearchBot) mostly see only the initial HTML, so client-rendered content can be invisible to them. If you can run a headless browser, do the diff; if not, flag "requires JS-rendering comparison — manual check" rather than silently skipping.
• Also compute a simple internal link-equity score (in/out degree per page from the crawl graph) to approximate an "internal link score" for money pages.

Phase 2 — On-page & Content

Across the sampled pages: title & description length/uniqueness (front-loading and relevance matter far more than hitting an exact character count — Google rewrites ~76% of titles), one H1 per page + logical H2–H6 nesting, image alt descriptiveness, slug hygiene, internal anchor text, freshness (lastmod / dateModified).

Run scripts/readability.py for Flesch Reading Ease + Flesch-Kincaid Grade. Detect page language first (<html lang>); these formulas are English-only, so skip or explicitly caveat readability for non-English pages rather than reporting a meaningless number.

Assess content depth vs. competitors, thin-content pages (< 300 words), scannability, and E-E-A-T signals — Trust is the most important pillar: author bios with verifiable credentials, clear sourcing, factual accuracy, extra YMYL scrutiny. Paraphrase content findings; don't reproduce the page's copy at length (see references/responsible-use.md).

Phase 3 — Performance & Core Web Vitals

If a Google Cloud API key is available, run PageSpeed Insights + CrUX for real Lighthouse lab scores and real-user field data (LCP / INP / CLS at p75). If not, estimate from the crawl: render-blocking scripts, image weight/format (WebP/AVIF? explicit dimensions?), caching/CDN response headers, third-party script count, font loading. Label the confidence: "scored from lab simulation only — provide a free Google Cloud API key for real-user CrUX data."

Thresholds (LCP ≤ 2.5s, INP ≤ 200ms, CLS ≤ 0.1 for "good") and the full method are in references/thresholds.md.

Phase 4 — UX & Accessibility (WCAG 2.2 AA)

If you have a code-execution sandbox, run axe-core or Pa11y locally (there is no free hosted WAVE API). Check contrast, ARIA, keyboard/focus, form labels, IA, mobile responsiveness. Use scripts/contrast.py to compute exact WCAG contrast ratios for suspect color pairs (4.5:1 normal text, 3:1 large text / UI components). Automated scanners catch only a subset — flag that manual review is still required for the rest.

Where a WCAG failure is one of the patterns commonly cited in ADA web-accessibility litigation, note that factually (not as legal advice) to reframe it from "nice-to-have" to "real risk," and suggest consulting counsel for genuine compliance assessment.

Phase 5 — CRO & Conversion (heuristic)

Scored as a heuristic — best-practice pattern presence, explicitly lower-confidence than the measured categories, because a pattern's presence isn't proof of conversion lift. Check: one clear primary CTA per key page, trust signals near conversion points, form-field friction, message match, value-prop clarity in first-screen copy, pricing clarity, risk reversal. Tailor funnel checks to the site type from Phase 0. Show your reasoning for qualitative judgments rather than emitting a bare number.

Phase 6 — Security, Analytics & AI-Readiness

• Security headers: run scripts/check_headers.py <url> to grade against the OWASP baseline (HSTS, CSP, X-Content-Type-Options, X-Frame-Options / frame-ancestors, Referrer-Policy, Permissions-Policy). Optionally call MDN HTTP Observatory v2 and SSL Labs (both free, no key) — see references/thresholds.md.
• Security self-scan (separate consent gate). The exposed-file probes (/.env, /.git/HEAD, /wp-config.php.bak, directory listing, /xmlrpc.php, user enumeration) and CMS/version disclosure checks are the most scanner-like behavior in the tool. Ask for a second, explicit confirmation before running scripts/security_scan.py, distinct from the Phase 0 gate. A 200 where you'd expect 403/404 is the finding. Owner-consented only.
• Analytics/tracking: detect GA4 / GTM / Meta Pixel from source, duplicate/conflicting tags, dataLayer presence, consent/cookie-banner (GDPR/CCPA) signals.
• AI Search Readiness / GEO: check robots.txt for AI-crawler directives (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, CCBot). Flag the common self-inflicted error of blocking the search/citation bots (OAI-SearchBot, Claude-SearchBot) while intending to block only training bots. Check direct-answer content structure (first 40–60 words of a section answer the implied question citably) and schema/entity consistency. Check /llms.txt presence and quality but frame it honestly — a low-cost bet for dev/API/agent-facing sites, not an SEO or AI-Overview-citation lever. Read references/geo-ai-readiness.md for the exact framing and the bot list.

Phase 7 — Scoring & Report

Compute the weighted 0–100 score using references/scoring-rubric.md (Technical SEO 18, On-page & Content 15, Performance 15, Accessibility 12, CRO 12, Security 10, GEO 8, Social/Local 5, Analytics 5). Three things stay outside the score, reported separately: off-page/backlinks (unverifiable for free), llms.txt (informational), and the AI Answer Accuracy check (qualitative). Next to each category sub-score, state its confidence level when it was estimated rather than measured.

Then generate the report using references/output-template.md:

1. Executive summary + overall score
2. Top 5 highest-impact fixes (sorted by points-recoverable ÷ effort)
3. Per-category scored sections: finding → measured evidence → benchmark → severity (P0–P3) → fix (with literal code where mechanical)
4. AI Search Readiness / GEO section
5. Appendix: checks that require paid tools or manual verification, with the exact Ahrefs/SEMrush/Moz/GSC report names to run

Visual language (apply consistently — this is what makes the report scannable). Use 🟢 / 🟡 / 🔴 for anything on a scale (category scores, a metric vs. its threshold, a section's overall state: 🟢 good/on-target, 🟡 needs-improvement/borderline, 🔴 poor/failing/urgent). Use ✅ / ❌ for binary facts (present/absent, correct/incorrect — HSTS present, exactly one H1, a broken link). Put the status symbol in the first column of every table and at the start of every finding line, one symbol system per column, so a reader can scan straight down the dots. Present the scorecard and the top-5 fixes as tables. references/output-template.md has the exact tables and the full symbol legend — follow it.

Differentiators to include when they fit

These turn a checklist into a deliverable. Read references/differentiators.md for the full spec. Apply the ones that fit the run — don't force all eight every time:

• Auto-generated fix code for every mechanical finding (the single biggest practical edge).
• Interactive visual report — a category radar chart + an impact-vs-effort quadrant, so quick wins are obvious. Deliver as an HTML artifact.
• Persistent trend tracking — store each run's score (total + per-category) via the artifact's private storage under audit:<domain>, and on re-run show the delta. Must use private (shared: false) storage only — never leak one user's audit history to strangers.
• Competitor comparison — repeat the pipeline for 1–3 competitor URLs, side-by-side scorecard.
• "What an AI crawler sees" — raw HTML vs. rendered page, flag material gaps.
• AI Answer Accuracy lite-check — with web_search, compare what AI answers say about the brand for 3–5 buyer-intent queries against what the site says. Explicitly a free one-time snapshot, not a substitute for paid AEO monitoring. Reported as findings, not points.
• Multi-audience output — a dev fix list, an executive one-pager, and a content-team checklist from the single run.

Honest limits — state these plainly, don't gloss

Backlinks, keyword rankings, and organic-traffic estimates cannot be reproduced for free — they require Ahrefs/SEMrush's proprietary web-scale indexes or the user's own GSC/analytics exports. Domain Rating / Authority Score are derived from the same data, so they're out too. Authenticated areas, non-HTML content (PDFs), and JS-only SPA content the environment can't render are out of scope — flag their presence rather than auditing their internals. Say all of this in the report's appendix; being upfront about the limit is itself a credibility signal.

Reference files

• references/thresholds.md — every hard-coded benchmark (CWV, meta, WCAG, security headers, social meta, images) + the free/no-auth API details and how to call them.
• references/scoring-rubric.md — the full weighted point table with per-sub-item breakdown.
• references/output-template.md — the exact report structure.
• references/schema-templates.md — required-field checklists + ready-to-paste JSON-LD blocks.
• references/geo-ai-readiness.md — AI-crawler bot list + the honest llms.txt framing.
• references/responsible-use.md — rules of engagement, consent gates, copyright, bot handling.
• references/differentiators.md — full spec for the eight differentiators.

For setup/troubleshooting, INSTALL.md is the full install guide (deps, the optional Google Cloud key, axe-core); scripts/check_env.py reports what's live in the current environment.