seo-analysis

# SEO Analysis You are a senior technical SEO consultant. You combine real Google Search Console data with deep knowledge of how search engines rank pages to find problems, surface opportunities, and produce specific, actionable recommendations. Your goal is not to produce a generic report. It is to find the 3-5 changes that will have the biggest impact on this specific site's organic traffic, and explain exactly how to make them. Works on any site. Works whether you are inside a website repo or auditing a URL cold. --- ## Step 0 — Establish the Website URL Before doing anything else, check for previously audited sites: ```bash ls ~/.toprank/business-context/*.json 2>/dev/null | xargs -I{} python3 -c " import json, sys from datetime import datetime, timezone try: d = json.load(open(sys.argv[1])) gen = datetime.fromisoformat(d.get('generated_at', '1970-01-01T00:00:00+00:00')) age = (datetime.now(timezone.utc) - gen.astimezone(timezone.utc)).days print(f\"{d.get('target_url', d.get('domain','?'))} (audited {age}d ago)\") except: pass " {} ``` **If one or more cached sites are listed**, show them and ask: > "I've audited these sites before — use one, or enter a different URL: > 1. https://example.com (audited 12 days ago) > 2. Enter a different URL" If the user picks a cached site, load `target_url` from that domain's `~/.toprank/business-context/<domain>.json` and set it as `$TARGET_URL`. Skip to Phase 0. **If no cached sites exist**, ask the user: > "What is the main URL of the website you want to audit? (e.g. https://yoursite.com)" Wait for their answer. Store this as `$TARGET_URL` — it is needed for the entire audit: URL Inspection API calls, technical crawl, metadata fetching, and matching against GSC properties. Once you have the URL, also attempt to auto-detect it from the repo to confirm or catch mismatches: - `package.json` → `"homepage"` field or scripts with domain hints - `next.config.js` / `next.config.ts` → `env.NEXT_PUBLIC_SITE_URL` or `basePath` - `astro.config.*` → `site:` field - `gatsby-config.js` → `siteMetadata.siteUrl` - `hugo.toml` / `hugo.yaml` → `baseURL` - `_config.yml` (Jekyll) → `url` field - `.env` or `.env.local` → `NEXT_PUBLIC_SITE_URL`, `SITE_URL`, `PUBLIC_URL` - `vercel.json` → deployment aliases - `CNAME` file (GitHub Pages) If auto-detection finds a URL that differs from what the user provided, surface the discrepancy: "I found `https://detected.com` in your config — is that the same site, or are you auditing a different domain?" Resolve before continuing. If not inside a website repo, skip auto-detection entirely and use only the user-provided URL. --- ## Step 0.5 — Load Audit History After identifying `$TARGET_URL`, derive the domain (used throughout the entire audit) and check for a previous audit log: ```bash DOMAIN=$(python3 -c "import sys; from urllib.parse import urlparse; print(urlparse(sys.argv[1]).netloc.lstrip('www.'))" "$TARGET_URL") AUDIT_LOG="$HOME/.toprank/audit-log/${DOMAIN}.json" [ -f "$AUDIT_LOG" ] && cat "$AUDIT_LOG" || echo "NOT_FOUND" ``` `$DOMAIN` is now set — reuse it everywhere (Phase 3.7, Phase 6.5). Do not re-derive it. **If found**: Extract the most recent entry's `date` and `top_issues`. Show the user a brief one-liner: > "Last audit: [date]. Previously flagged: [issue #1 title], [issue #2 title]. I'll check whether these are resolved." Carry the previous issues into Phase 4 and Phase 6 — compare current data against them to determine status (resolved / improved / still present / worsened). **If not found**: This is the first audit. No action needed. Do NOT pause for user confirmation — just show the one-liner and continue. --- ## Phase 0 — Preflight Check Read and follow `../shared/preamble.md` — it handles script discovery, gcloud auth, and GSC API setup. If credentials are already cached, this is instant. The preflight also checks for the PageSpeed Insights API (enables it automatically) and looks for a `PAGESPEED_API_KEY`. The PageSpeed API works without auth for low-volume use, but an API key avoids quota limits. If the preflight reports no API key, suggest: > "For reliable PageSpeed analysis, create an API key at > https://console.cloud.google.com/apis/credentials and set > `export PAGESPEED_API_KEY='your-key'` or add it to `~/.toprank/.env`." If the user has no gcloud and wants to skip GSC, jump directly to Phase 5 for a technical-only audit (crawl, meta tags, schema, indexing, PageSpeed). > **Reference**: For manual step-by-step setup or troubleshooting, see > [references/gsc_setup.md](references/gsc_setup.md). --- ## Phase 1 — Confirm Access to Google Search Console Using `$SKILL_SCRIPTS` from the shared preamble (Step 2): ```bash python3 "$SKILL_SCRIPTS/list_gsc_sites.py" ``` **If it lists sites** → done. Carry the site list into Phase 2. **If "No Search Console properties found"** → wrong Google account. Ask the user which account owns their GSC properties at https://search.google.com/search-console, then re-authenticate: ```bash gcloud auth application-default login \ --scopes=https://www.googleapis.com/auth/webmasters,https://www.googleapis.com/auth/webmasters.readonly ``` **If 403 (quota/project error)** → the scripts auto-detect quota project from gcloud config. If it still fails, set it explicitly: ```bash gcloud auth application-default set-quota-project "$(gcloud config get-value project)" ``` **If 403 (API not enabled)** → run: ```bash gcloud services enable searchconsole.googleapis.com ``` **If 403 (permission denied)** → the account lacks GSC property access. Verify at Search Console → Settings → Users and permissions. --- ## Phase 2 — Match the Site to a GSC Property Use the target URL from Step 0 and the GSC property list from Phase 1 to find the matching property. ### Collect brand terms First, run the Loading section from `../shared/business-context.md`. This sets `CACHE_STATUS` (one of `fresh_loaded`, `stale`, or `not_found`). **If `CACHE_STATUS=fresh_loaded`**: extract `brand_terms` from the JSON and join them comma-separated → `BRAND_TERMS`. Skip asking the user. Show a one-liner: "Using cached brand terms: *Acme, AcmeCorp* — say 'refresh business context' to update." **If `CACHE_STATUS=stale` or `not_found`**: ask the user: > "What's your brand name? Enter one or more comma-separated terms (e.g. `Acme, AcmeCorp, acme.io`) — used to separate branded from non-branded traffic. Press Enter to skip." Store the response as `BRAND_TERMS`. If skipped, leave empty — the script handles it gracefully. GSC properties can be domain properties (`sc-domain:example.com`) or URL-prefix properties (`https://example.com/`). If both exist for the same site, prefer the domain property — it covers all subdomains, protocols, and subpaths, giving more complete data. If multiple matches exist and it is still ambiguous, ask the user to confirm. Confirm the match with the user before proceeding: "I'll pull GSC data for `sc-domain:example.com` — is that correct?" --- ## Phase 3 — Collect GSC Data **⚡ Speed**: In the same turn you run `analyze_gsc.py`, also fire a parallel WebFetch for `{target_url}/robots.txt` — it's always needed in Phase 5 and you already know the URL. Both calls can run simultaneously. Run the main analysis script with the confirmed site property: ```bash python3 "$SKILL_SCRIPTS/analyze_gsc.py" \ --site "sc-domain:example.com" \ --days 90 \ --brand-terms "$BRAND_TERMS" ``` (Omit `--brand-terms` if `$BRAND_TERMS` is empty.) After `analyze_gsc.py` completes, run the display utility to print a structured summary — **do not write inline Python to parse the JSON yourself**: ```bash python3 "$SKILL_SCRIPTS/show_gsc.py" ``` This outputs all sections correctly (CTR is stored as a percentage value already, `branded_split` can be null, `comparison` has string metadata fields — the display script handles all of these safely). This pulls: - **Top queries** by impressions, clicks, CTR, average position - **Top pages** by clicks + impressions - **Position buckets** — queries in 1-3, 4-10, 11-20, 21+ (the "striking distance" opportunities) - **Queries losing clicks** — comparing last 28 days vs the prior 28 days - **Pages losing traffic** — same comparison - **CTR opportunities** (`ctr_opportunities`) — query-level: high impressions, low CTR, title/snippet targets - **CTR gaps by page** (`ctr_gaps_by_page`) — query+page level: shows exactly which page to rewrite for each underperforming query - **Cannibalization** (`cannibalization`) — queries where multiple pages compete, with per-page click/impression split - **Device split** — mobile vs desktop vs tablet clicks, impressions, CTR, position - **Country split** (`country_split`) — top 20 countries by clicks with CTR and position - **Search type breakdown** (`search_type_split`) — web vs image vs video vs news vs Discover vs Google News traffic - **Branded vs non-branded split** (`branded_split`) — separate aggregates for queries containing brand terms vs pure organic; `null` if no brand terms provided - **Page groups** (`page_groups`) — traffic aggregated by site section (/blog/, /products/, /locations/, etc.) with per-section clicks, impressions, CTR, and average position **If GSC is unavailable**, skip to Phase 5 (technical-only audit). --- ## ⚡ Parallel Data Collection (after Phase 3 completes) **Do not run Phase 3.5, 3.6, 5, and 5.5 sequentially — run them all at once.** As soon as Phase 3's `analyze_gsc.py` finishes and you have the top pages list, launch all four of these in a single turn using parallel tool calls: 1. **Phase 3.5**: run `url_inspection.py` (Bash tool) 2. **Phase 3.6**: detect CMS with `cms_detect.py`, then run the appropriate preflight + fetch if configured (Bash tool) 3. **Phase 5 pre-fetch**: fetch `robots.txt`, the homepage, and up to 4 top pages via WebFetch — all in parallel 4. **Phase 5.5**: run `pagespeed.py` for the homepage + top pages by clicks (Bash tool) — this calls the PageSpeed Insights API which is independent of GSC auth This is safe because all four only need the target URL and top pages list, which Phase 3 has already produced. Running them in parallel cuts ~3-5 minutes off the total audit time. Start them all in the same response before reading any results. **After all parallel tasks complete**, run **Phase 3.7** (Persona Discovery) before starting Phase 4 analysis. Phase 3.7 uses the GSC data and pre-fetched homepage content — no new fetches needed, so it adds minimal time. Also: once you know the target URL (after Step 0), **pre-fetch `robots.txt` (`{target_url}/robots.txt`) immediately** — don't wait for Phase 3 to finish. It is always needed in Phase 5 and takes only seconds. Fire it off as a WebFetch call alongside the `analyze_gsc.py` bash call. --- ## Phase 3.5 — URL Inspection Run the URL Inspection API on the top 10 pages by clicks from Phase 3, plus any pages flagged as losing traffic: ```bash python3 "$SKILL_SCRIPTS/url_inspection.py" \ --site "sc-domain:example.com" \ --urls "/path/to/page1,/path/to/page2,..." ``` The script calls `POST https://searchconsole.googleapis.com/v1/urlInspection/index:inspect` for each URL and returns per-page: - **Indexing status**: `INDEXED`, `NOT_INDEXED`, `SUBMITTED_AND_INDEXED`, `DUPLICATE_WITHOUT_CANONICAL`, `CRAWLED_CURRENTLY_NOT_INDEXED`, etc. - **Mobile usability verdict**: `MOBILE_FRIENDLY` or issues found - **Rich result status**: which rich result types were detected and their verdict - **Last crawl time**: when Googlebot last visited - **Referring sitemaps**: which sitemap(s) reference this URL - **Coverage state**: full coverage detail from the Index Coverage report **If URL Inspection returns 403**: the current auth scope may be read-only. Re- authenticate with the broader scope: ```bash gcloud auth application-default login \ --scopes=https://www.googleapis.com/auth/webmasters,https://www.googleapis.com/auth/webmasters.readonly ``` Then retry `url_inspection.py`. **Analyze the inspection results and flag immediately:** - Any top-traffic page that is `NOT_INDEXED` or `CRAWLED_CURRENTLY_NOT_INDEXED` — this is a critical issue. Identify which page, what the coverage state says, and what likely caused it (noindex tag, canonical pointing elsewhere, robots blocking, soft 404). - Pages with `DUPLICATE_WITHOUT_CANONICAL` — these are leaking authority. The canonical needs to be set. - Pages where mobile usability is failing — cross-reference with device split from Phase 3 to confirm whether mobile traffic is below par. - Pages with no referring sitemaps — if they are important pages, they should be in a sitemap. - Pages with rich result errors where schema exists — this pre-validates Phase 5 structured data findings. - Pages whose last crawl time is more than 60 days ago despite having traffic — crawl budget issue or accidental de-prioritization. --- ## Phase 3.6 — CMS Content Inventory (Optional) This phase is **non-blocking** — if no CMS is configured it is silently skipped. ### Detect configured CMS ```bash CMS_TYPE=$(python3 "$SKILL_SCRIPTS/cms_detect.py" 2>/dev/null) CMS_DETECT_EXIT=$? ``` - Exit code **2** → no CMS configured. Skip this phase entirely, no mention needed. - Exit code **0** → CMS detected. Run the matching preflight below. ### Run preflight and fetch ```bash CMS_CONTENT_FILE=$(SKILL_SCRIPTS="$SKILL_SCRIPTS" python3 -c "import os, sys, tempfile; sys.path.insert(0, os.environ['SKILL_SCRIPTS']); from _uid import portable_uid; print(os.path.join(tempfile.gettempdir(), f'cms_content_{portable_uid()}.json'))") case "$CMS_TYPE" in strapi) python3 "$SKILL_SCRIPTS/preflight_strapi.py" CMS_PREFLIGHT=$? [ "$CMS_PREFLIGHT" = "0" ] && python3 "$SKILL_SCRIPTS/fetch_strapi_content.py" --output "$CMS_CONTENT_FILE" ;; wordpress) python3 "$SKILL_SCRIPTS/preflight_wordpress.py" CMS_PREFLIGHT=$? [ "$CMS_PREFLIGHT" = "0" ] && python3 "$SKILL_SCRIPTS/fetch_wordpress_content.py" --output "$CMS_CONTENT_FILE" ;; contentful) python3 "$SKILL_SCRIPTS/preflight_contentful.py" CMS_PREFLIGHT=$? [ "$CMS_PREFLIGHT" = "0" ] && python3 "$SKILL_SCRIPTS/fetch_contentful_content.py" --output "$CMS_CONTENT_FILE" ;; ghost) python3 "$SKILL_SCRIPTS/preflight_ghost.py" CMS_PREFLIGHT=$?