seo

# Steel SEO Pulse A repeatable **monthly** SEO check for Steel's two web properties — **docs.steel.dev** (docs) and **steel.dev** (marketing). It answers "is the SEO work paying off, and what moved since last month?" by combining real DataForSEO keyword + SERP data with a live technical audit of the key pages, then writing a dated delta report. ## What it produces Each run writes, under `scripts/seo/pulse/`: - **Snapshots** (`snapshots/volumes-YYYY-MM.json`, `ranks-YYYY-MM.json`, `tech-YYYY-MM.json`, `perf-YYYY-MM.json`) — the raw state for this month. Next month's run diffs against these. - **`reports/seo-YYYY-MM.md`** — the human-readable delta report (the deliverable). Snapshots are gitignored (regenerable state); reports are kept for history. ## Prerequisites - **DataForSEO credentials** in the repo-root `.env`: `DATAFORSEO_AUTH=<base64 login:password>` (plus `DATAFORSEO_LOGIN`/`DATAFORSEO_PASSWORD`). If `.env` lacks them, ask the user to add them before proceeding. - The account must be **email-verified** and have a positive balance (SERP calls cost credits — see Cost below). - Run all scripts from the **repo root** (`/home/agent/steel-docs`). ## How to read the numbers (important) The keyword data comes from DataForSEO's `keywords_data/google_ads/search_volume/live`. Understand its limits so you interpret it correctly: - It's **Google Ads planner data**: US-only, **bucketed**, and it **under-reports low-volume developer terms**. A `volume` of `10` or a row with `volume: null` means "below the ads reporting threshold / no advertisers" — **not zero demand**. Plenty of real organic traffic lives at `10`/`null`. - `paid_competition` / `competition_index` is **paid-search competition, not organic ranking difficulty.** Don't treat "LOW competition" as "easy to rank." - `cpc` is a great **commercial-intent signal**: a high CPC on a modest-volume term (e.g. `openai agents sdk` ~$43, `best scraping api` ~$46) flags strong buyer intent worth pursuing. - The `trend` field holds the prior 12 months of monthly volume — use it to spot **momentum** (a term spiking is often more valuable than a high-but-flat one). So: weight **volume × CPC × momentum** together, and treat the numbers as directional, not exact. ### Impression sinks: always report an ex-sinks view A couple of pages pull huge impression volumes at near-zero CTR, so any site-level clicks/impressions/CTR figure that includes them describes those pages instead of the docs. They live in `config.json` → `impression_sinks` (with the reason each one is listed). **Whenever you report a site-level click, impression, or CTR number, report the ex-sinks figure next to it** and say which pages were excluded. Never delete the as-exported number: keep both, so a shift in the sinks themselves stays visible. ## The monthly workflow Run the deterministic scripts in order (they print summaries and write snapshots), then add judgment for drift/discovery, then synthesize the report. The split is deliberate — **scripts handle the mechanical, repeatable data; you handle synthesis and judgment.** (A prior attempt to do everything in one big agent fan-out got stuck in loops; scripts are reliable and cheap to re-run.) ### Phase 0: Search performance (needs a Search Console export) ```bash node .claude/skills/seo/scripts/gsc-perf.mjs <path-to-gsc-pages-export.csv> ``` Ask the user for a Search Console **Pages** export (docs.steel.dev, last 3 months, CSV). The script prints site clicks/impressions/CTR **as exported and ex-sinks**, breaks out each `impression_sinks` page with its share of site impressions, writes `perf-YYYY-MM.json`, and prints last month's ex-sinks totals for comparison. Skip the phase if no export is available and say so in the report: don't quote site CTR from memory. ### Phase 1 — Refresh demand ```bash node .claude/skills/seo/scripts/dfs-volumes.mjs ``` Pulls volume/CPC/12-month trend for every keyword in `config.json` → `tracked_keywords`, writes `volumes-YYYY-MM.json`, and prints **movers** (Δvolume ≥50 or trend slope ≥25% vs last month). For a cheap smoke test add `--limit 10`. ### Phase 2 — Track SERP positions ⚠️ costs credits ```bash node .claude/skills/seo/scripts/dfs-rank.mjs # budget tight? → node dfs-rank.mjs --limit 10 ``` Checks Google rank (top 20) for `docs.steel.dev` **and** `steel.dev` on the **ownable** terms in `config.json` → `rank_keywords` (not head terms — Steel won't displace Wikipedia/official docs on "selenium", so tracking it wastes money). Prints rank changes vs last month. If you hit `40200 Payment Required`, the DataForSEO balance is exhausted — stop, tell the user to top up at app.dataforseo.com, and note in the report that rank tracking was partial. ### Phase 3 — Technical SEO audit ```bash node .claude/skills/seo/scripts/tech-audit.mjs ``` Fetches every page in `config.json` → `key_pages`, checks status, title length, description length, canonical correctness, JSON-LD presence, accidental `noindex`, and whether each P0 page's **title still carries its target intent phrase** (`target_any`). Prints: which P0 titles are **still generic** (the title fix hasn't shipped yet — this is the main progress tracker), short/missing descriptions, and any regressions vs last month (broken page, lost canonical/JSON-LD, new noindex). No DataForSEO cost — pure live fetches. ### Phase 4 — Drift & discovery (your judgment) The scripts can't see emerging demand or competitor moves; you can. Spend a few minutes: - Scan the **movers** from Phase 1 — which terms are *rising*? In the fast-moving AI-agent space (computer-use, new frameworks), a spiking term is a leading indicator of a page worth building. - For 3–5 top ownable terms, run `WebSearch` and note **SERP drift**: new competitor "vs"/alternative pages, new listicles, or a Steel page that quietly started ranking. - Propose **new keywords/pages** to add to `config.json` next month. ### Phase 5 — Synthesize the report Read the three script summaries + your Phase 4 notes and write `scripts/seo/pulse/reports/seo-YYYY-MM.md` (create `reports/` if missing). Use this template: ```markdown # Steel SEO Pulse — YYYY-MM **vs previous:** YYYY-MM · **generated:** YYYY-MM-DD ## TL;DR 2–4 sentences: did the P0 title fixes ship / move rankings? Any regressions? Biggest new opportunity? ## Search performance | View | Clicks | Impressions | CTR | |---|---|---|---| | Site (as exported) | … | … | … | | Site (ex-sinks) | … | … | … | Ex-sinks excludes `config.json` → `impression_sinks`; list the excluded pages with their impressions and CTR. Compare month over month on the **ex-sinks** row. ## Demand movers - 📈 Rising: <term> (prev → now, +X% trend) — why it matters / suggested action - 📉 Falling: <term> … ## Rankings (ownable terms) - Table of docs.steel.dev / steel.dev positions, bolding changes vs last month (↑ improved / ↓ declined / new / dropped). - Note if rank tracking was partial due to DataForSEO balance. ## Technical health - Title optimization progress: X/N P0 pages now optimized; still-generic list (with URLs). - New regressions (broken pages, lost canonical/JSON-LD, new noindex) — or "none." - Description fixes still owed (Phase 3 flags short/missing meta descriptions). ## Drift & discovery - New competitor pages / SERP changes observed. - Emerging keywords to add to tracking next month. ## Recommended actions this month 3–6 concrete, prioritized items (impact × effort), tagged docs vs marketing. ``` Keep it tight and skimmable — the reader is busy. Lead with what changed, not a re-explanation of the basics. ## The prioritization framework (reuse when recommending actions) When the report calls for new work, rank opportunities by **impact × effort**, and use the lenses from the original `SEO-OPPORTUNITIES.md`: - **Volume × CPC × momentum** (Phase 1 data) for demand. - **Intent × rankability**: prefer *informational / problem-solving / framework-tutorial / comparison* queries (docs can own these) over pure buyer/pricing queries. - **Existing-asset leverage**: optimizing a page Steel already has (e.g. a generic-titled integration page behind a 110k/mo term) beats building from scratch. - **Ownership**: buyer/pricing/comparison-landing intent → **steel.dev marketing**; tutorial/concept/troubleshooting/migration intent → **docs.steel.dev**. Don't recommend docs pages for things the marketing site must own. ## Config & maintenance All tracked keywords, key pages, and competitors live in `.claude/skills/seo/config.json`. To maintain tracking quality: - Add newly-discovered ownable terms to `tracked_keywords` (and `rank_keywords` if worth tracking position). - Add a `key_pages` entry (with `target_any` = the intent phrase the title should carry) whenever a new high-value page is published, so the audit tracks whether its title gets optimized. - Remove terms that have gone stale. - Add a page to `impression_sinks` once it pulls large impression volumes at near-zero CTR (and remove it when it stops), so the ex-sinks view keeps tracking real docs performance. ## Cost awareness - **Volumes**: ~$0.02/run (45 keywords, batched 25/call). Negligible — run freely. - **SERP ranks**: ~$0.05–0.10/keyword → ~$2–3 for the full `rank_keywords` set. This is the costly phase and the account balance is finite. Use `--limit`, and if you see `40200`, stop and flag it. If budget is a recurring constraint, trim `rank_keywords` to the ~10 terms that matter most. - **Tech audit**: free (live fetches only). - **Search performance**: free (parses a Search Console CSV export). If DataForSEO is unavailable (not verified, or balance exhausted), you can still run Phase 3 (free) and do a qualitative Phase 4 via `WebSearch` — note the gap in the report rather than skipping the whole pulse.