competitor-analysis

# Competitor Analysis ## When to Use Use when the user needs structured competitor research with Browserbase discovery, enrichment lanes, screenshots, comparison matrices, and a final HTML report. _Source: [browserbase/skills](https://github.com/browserbase/skills) (MIT)._ Analyze a user's competitors. Uses Browserbase Search API for discovery and a 4-lane Plan→Research→Synthesize pattern for enrichment — outputting an HTML report with overview, per-competitor deep dives, a side-by-side feature/pricing matrix, and a chronological mentions feed. **Required**: `BROWSERBASE_API_KEY` env var and the `browse` CLI installed (`npm install -g browse`). **First-run setup**: On the first run you'll be prompted to approve `browse cloud fetch`, `browse cloud search`, `cat`, `mkdir`, `sed`, etc. Select **"Yes, and don't ask again for: browse cloud fetch:\*"** (or equivalent) for each. To permanently approve, add these to your `~/.claude/settings.json` under `permissions.allow`: ```json "Bash(browse:*)", "Bash(bunx:*)", "Bash(bun:*)", "Bash(node:*)", "Bash(cat:*)", "Bash(mkdir:*)", "Bash(sed:*)", "Bash(head:*)", "Bash(tr:*)", "Bash(rm:*)" ``` **Path rules**: Always use full literal paths in Bash — NOT `~` or `$HOME`. Resolve the home directory once and use it everywhere. When building subagent prompts, replace `{SKILL_DIR}` with the full literal path. **Output directory**: All output goes to `~/Desktop/{company_slug}_competitors_{YYYY-MM-DD}/`. This directory contains one `.md` file per competitor plus the generated HTML views and CSV. **CRITICAL — Tool restrictions (applies to main agent AND all subagents)**: - All web searches: use `browse cloud search`. NEVER WebSearch. - All page fetches: use `browse cloud fetch --allow-redirects` (returns markdown by default; add `--format raw` if you need the original HTML, then pipe through `sed ... | tr -s ' \n'` to extract text). NEVER WebFetch. 1 MB response limit — fall back to `browse get markdown` (after `browse open <url> --remote`) for JS-heavy pages. - All research output: subagents write **one markdown file per competitor** to `{OUTPUT_DIR}/{competitor-slug}.md` using bash heredoc. NEVER use the Write tool or `python3 -c`. See `references/example-research.md` for the file format. - Report compilation: use `node {SKILL_DIR}/scripts/compile_report.mjs {OUTPUT_DIR} --user-company "{user_company}" --open` — generates `index.html`, `competitors/*.html`, `matrix.html`, `mentions.html`, `results.csv` in one step and opens overview. - URL deduplication: `node {SKILL_DIR}/scripts/list_urls.mjs /tmp --prefix competitor`. - **Subagents must use ONLY the Bash tool.** - **Main agent NEVER reads raw discovery JSON batch files.** **CRITICAL — Minimize permission prompts**: - Subagents MUST batch ALL file writes into a SINGLE Bash call using chained heredocs. - Batch ALL searches and ALL fetches into single Bash calls via `&&` chaining. ## Pipeline Overview Follow these 8 steps in order. Do not skip or reorder. 1. **User Company Research** — Deeply understand the user's company, produce `precise_category` + `category_include_keywords` + `exclusion_list` 2. **Depth Mode + Seed Input** — Choose depth, accept optional seed competitor URLs 3. **Discovery (3 parallel waves)** — Wave A (alternatives), Wave B (precise category), Wave C (comparison-page graph via "X vs Y" title parsing) 4. **Gate** — `scripts/gate_candidates.mjs` fetches each candidate's hero text (via `browse cloud fetch`) and drops wrong-category URLs 5. **Confirm enrichment set with the user** — Present PASS / UNKNOWN / rejected-brand-matches via `AskUserQuestion`. User ticks the real ones, adds any the discovery missed. Skipping this step is wasteful because enrichment is expensive (25 subagents × depth budget) and the gate is imperfect (JS-heavy homepages, Cloudflare challenges, semantic-variant taglines) 6. **Deep Enrichment (5 subagents per competitor in deep/deeper modes)** — Marketing, Discussion, Social, News, Technical — each lane a separate subagent writing to `partials/`; then `merge_partials.mjs` consolidates. In deep/deeper modes, **Step 5d** adds a 6th Battle Card synthesis lane AFTER Step 5c fact-check completes — produces per-competitor Landmines / Objection Handlers / Talk Tracks grounded in cited evidence. 7. **Screenshots** — `capture_screenshots.mjs` via the `browse` CLI captures a 1280×800 homepage hero per competitor 8. **HTML Report** — Overview + per-competitor (with embedded hero screenshot + Battle Card card) + matrix + mentions views --- ## Step 0: Setup Output Directory ```bash OUTPUT_DIR=~/Desktop/{company_slug}_competitors_{YYYY-MM-DD} mkdir -p "$OUTPUT_DIR" ``` Replace `{company_slug}` with the user's company name (lowercase, hyphenated) and `{YYYY-MM-DD}` with today's date. Pass `{OUTPUT_DIR}` as a full literal path to every subagent. Clean up discovery batch files from prior runs: ```bash rm -f /tmp/competitor_discovery_batch_*.json ``` **Re-runs must start from a clean `$OUTPUT_DIR`.** `compile_report.mjs` ingests *every* `{slug}.md` in the directory, and `merge_partials.mjs` only overwrites the slugs in the current set — it never deletes ones dropped from a new enrichment set. Since the directory is keyed by date, a same-day re-run with a different competitor set would leave stale competitors in the overview, matrix, CSV, and screenshots. Either use a fresh directory or clear the prior per-competitor files first: ```bash rm -f "$OUTPUT_DIR"/*.md && rm -rf "$OUTPUT_DIR"/partials "$OUTPUT_DIR"/screenshots ``` ## Step 1: User Company Research This step sets the baseline for what "competitor" means AND produces the verified data the Step 5b matrix will use for the `userCompany` row. **Rule**: The user's company gets the same 5-lane research depth as competitors. Do NOT fill `userCompany` in matrix.json from memory — it will ship false claims to the user's own team. On a search-API run (user company Exa, 2026-04-23), skipping this step produced a matrix that claimed Exa had a "published uptime SLA" (there is no numeric public SLA — only a status page) and marked its MIT-licensed Python SDK as `open-source: false` (the repo is github.com/exa-labs/exa-py, LICENSE confirmed MIT). Both errors would have surfaced in the "Where you're winning" card as fabricated moats. Process: 1. Ask the user for their company name or URL. 2. **Check for an existing profile** at `{SKILL_DIR}/profiles/{company-slug}.json`. If it exists, load it and confirm with the user: "I have your profile from {researched_at}. Still accurate?" — if yes, skip to Step 2 BUT still run the partial-lane enrichment below so matrix synthesis has fresh feature evidence. The profile format is shared with `company-research` (same shape). If a user already has a profile saved under `company-research/profiles/`, you may copy it into this skill's profiles directory rather than re-researching. 3. **Run the full 5-lane enrichment on the user's company** — identical to the competitor pattern in Step 5. For each lane, spawn a Bash-only subagent that writes to `{OUTPUT_DIR}/partials/{user-slug}.{lane}.md`: - **marketing** — tagline, positioning, pricing tiers, features, integrations, open-source components (SDK repos + licenses), regions offered, compliance (SOC 2 / HIPAA / trust portal URL) - **technical** — REST + streaming API support (with docs URLs), SDK languages, MCP server URL, neural vs keyword retrieval modes, reranking / highlights / live-crawl specifics, published uptime SLA (actual %, not status page), third-party retrieval-quality benchmarks - **discussion**, **social**, **news** — optional in quick mode, recommended in deep+ See `references/research-patterns.md` → "Self-Research" for sub-questions. Each finding MUST cite a URL. 4. Run `merge_partials.mjs` on the user's partials too — produces `{OUTPUT_DIR}/{user-slug}.md`, the canonical source Step 5b reads from for `userCompany` flags. 5. Synthesize into a profile: Company, Product, Existing Customers, Competitors (seed list), Use Cases, **precise_category**, **category_include_keywords**, **exclusion_list**. Do NOT include ICP — this skill doesn't need it. - `precise_category`: one sentence describing the category. e.g., "AI web search API for agents with neural + keyword retrieval". Avoid vague words like "tools" / "platform". - `category_include_keywords`: 8-15 phrases a direct competitor's marketing would likely contain (hero or title). Include semantic variants. - `exclusion_list`: phrases that indicate a *different* category — used by the gate to reject false positives (e.g. `antidetect browser`, `scraping api`, `screenshot api`, `residential proxy`). See `references/research-patterns.md` → "Synthesis Output" for the exact format and Exa as a worked example. 6. Present the profile + the user-company `.md` to the user for confirmation. Do not proceed until confirmed. 7. **Save the confirmed profile** to `{SKILL_DIR}/profiles/{company-slug}.json`. ## Step 2: Depth Mode + Seed Input Ask clarifying questions via `AskUserQuestion` with checkboxes: - **Known competitors?** Text area for URLs/names (optional — discovery will find more). - **Depth mode?** - `quick` — marketing surface only, many competitors, ~2-3 tool calls each - `deep` — + external signal (mentions, reviews, news), ~5-8 tool calls each - `deeper` — + public benchmarks + strategic diff vs user's company, ~10-15 tool calls each - **Target count?** Rough number of competitors to research (e.g., 10 / 20 / 50). This is the ONLY user interaction. After this, execute silently until the report is ready. | Mode | Research per competitor | Best for | |------|--------------------------|----------| | `quick` | Lane 1 only (homepage + pricing) | Scanning ~30-50 competitors fast | | `deep` | Lanes 1+2 | ~15-25 competitors with external signal | | `deeper` | All 4 lanes (+ benchmarks + strategic diff) | ~5-15 competitors with full intel | ## Step 3: Discovery (3 parallel waves) **Formula**: `ceil(target_count / 20)` queries per wave. Over-discover ~3x because the gate drops ~40-60%. Evaluation on a search-API run shows all three waves are additive — skip any and you lose real competitors: **Wave A — Generic alternatives** (broad; heavy aggregator noise, filtered out later) - `"alternatives to {user_company}"` - `"{user_company} competitors"` **Wave B — Precise category** (uses `precise_category` from the profile) - `"{precise_category}"` verbatim - 2-3 queries composed from the most distinctive tokens (e.g. `"web search api for ai agents"`, `"retrieval API for LLMs"`) **Wave C — Comparison-page graph** (highest precision) - `"{user_company} vs"` - `"{seed1} vs"`, `"{seed2} vs"`, `"{seed3} vs"` (seeds from the profile's `competitors` list) - After the searches, run `scripts/extract_vs_names.mjs` to parse `"X vs Y"` patterns from result titles — this uniquely surfaces competitors that don't appear as URL hits. **Process**: 1. Issue **3 parallel `browse cloud search` Bash calls** (one per wave) in a SINGLE message — NOT subagents. Each Bash call chains its 2-4 queries with `&&`. See `references/workflow.md` → "Discovery — parallel Bash, not subagents" for the exact recipe. Subagents are too heavy for a workload of 6-12 `browse cloud search` calls. 2. After all waves complete: ```bash node {SKILL_DIR}/scripts/list_urls.mjs /tmp --prefix competitor > /tmp/competitor_urls.txt node {SKILL_DIR}/scripts/extract_vs_names.mjs /tmp --prefix competitor \ --seed "{user_company},{seed1},{seed2},{seed3}" \ > /tmp/competitor_vs_names.jsonl ``` 3. **Filter** `/tmp/competitor_urls.txt` — remove blog posts, news, AI-tool directories (seektool.ai, respan.ai, agentsindex.ai, toolradar.com, aitoolsatlas.ai, vibecodedthis.com, etc.), review aggregators (g2.com, capterra.com), databases (crunchbase.com, tracxn.com), user's own domain. See `references/workflow.md` for the full noise-domain list. 4. For `vs_names` entries that have a resolved `domain`, add them. For unresolved names, optionally run `browse cloud search "{name}" --num-results 3` and pick the top root domain. 5. Merge with user-provided seed URLs. Dedup by hostname → `/tmp/competitor_candidates.txt`. ## Step 4: Gate (category-fit filter) Drop candidates whose marketing identifies them as a *different* category before enrichment burns tool calls on them. ```bash cat /tmp/competitor_candidates.txt \ | node {SKILL_DIR}/scripts/gate_candidates.mjs \ --include "{profile.category_include_keywords joined with commas}" \ --exclude "{profile.exclusion_list joined with commas}" \ --concurrency 6 \ > /tmp/competitor_gated.jsonl grep '"status":"PASS"' /tmp/competitor_gated.jsonl \ | node -e 'require("fs").readFileSync(0,"utf-8").split("\n").filter(Boolean).forEach(l => { try { console.log(JSON.parse(l).url); } catch {} })' \ > /tmp/competitor_passed.txt ``` The gate fetches each candidate's homepage via `browse cloud fetch --allow-redirects --format raw`, extracts the first 800 chars of visible text, and classifies position-aware: exclude in `<title>` → REJECT; include in `<title>` → PASS; hybrid title → hero200 tiebreak; otherwise fall through. **Evaluated on a search-API run** with 12 mixed candidates: 7/7 real competitors passed, 4/4 wrong-category rejected, 1 known-hybrid edge case rejected. ## Step 4.5: Confirm enrichment set with the user **This step is mandatory. Do NOT skip to enrichment just because the gate ran.** Enrichment is expensive: 5 competitors × 5 lane-subagents = 25 subagents, ~10-15 minutes of wall clock, ~300 `browse cloud` calls. Running it on the wrong set wastes all of that. The gate also has known blind spots: - **JS-heavy homepages** (e.g. Tavily, Firecrawl) — `browse cloud fetch` returns near-empty text, so keyword matching has nothing to match on → REJECT or UNKNOWN - **Cloudflare challenge pages** (e.g. Perplexity) — title becomes "Just a moment..." → no category signal - **Semantic variants** — "search foundation" / "retrieval backbone" don't lexically match a list centered on "search API" - **Domain ambiguity** — `brave.com` (the browser) vs `api-dashboard.search.brave.com` (the actual API product) can confuse classification The user almost always has domain knowledge the skill lacks. Ask them. **Process** — the main agent: 1. Read `/tmp/competitor_gated.jsonl` and group rows: - **PASS bucket**: everything with status=PASS. - **UNKNOWN bucket**: status=UNKNOWN (fetch failed — always surface, these are the silent misses). - **Rejected-brand bucket**: top ~10 REJECT rows whose title mentions a well-known brand pattern (e.g. contains the token from a user-supplied seed list, or appears frequently in the Wave C "X vs Y" graph). 2. Present the buckets to the user, one table per bucket, with URL + title + reason (for rejects). 3. Use `AskUserQuestion` with a checkbox list of all candidates across the three buckets, plus a free-text "add more" field. The prompt should be explicit: > "Here are the gate's picks plus a few it was unsure about. Tick the ones that are real competitors in your space, and paste any URLs I missed (comma-separated). Enrichment will run on ONLY the ticked set." 4. Write the confirmed set to `/tmp/competitor_enrichment_set.txt` (one URL per line). This is the input for Step 5 — not `/tmp/competitor_passed.txt`. **If the user doesn't respond** or explicitly says "just run it", fall back to `/tmp/competitor_passed.txt` as-is, but warn in chat that the run may waste budget on wrong-category hits.