browser-cli-site-guide

name	browser-cli-site-guide
description	Create browser-cli site-specific guides by exploring live website DOM with browser-cli. Use when asked to create, add, or write a new site guide (site reference, site-specific skill) for browser-cli. Follows a structured workflow: navigate site pages, discover real CSS selectors from the live DOM, build and validate extraction scripts interactively, then write a tested guide file. Triggers on: "add a site guide for github.com", "create browser-cli reference for reddit", "write selectors for youtube.com", "make a new site guide", "add browser-cli support for X", "create a scraper for Y", "write extraction scripts for Z". Also trigger when the user mentions wanting to automate data extraction from a specific website that doesn't have an existing guide.
allowed-tools	Bash(browser-cli:*), Bash(cat *), Bash(mktemp *), Bash(TMP=*)
argument-hint	<domain to create guide for, e.g. github.com>

Site Guide Creator

Create tested site-specific guides for browser-cli. Each guide has two deliverables:

1. Selector reference (references/sites/<domain>.md) — CSS selectors, URL patterns, extraction commands
2. Recipe script (scripts/<domain-short>.mjs) — reusable functions for common operations

Both are verified against the live DOM before writing.

Prerequisites

Read an existing guide + script pair to understand the conventions before starting:

• Simple site: references/sites/news.ycombinator.com.md + scripts/hn.mjs
• Complex site: references/sites/google.com.md + scripts/google.mjs
• data-testid heavy: references/sites/x.com.md + scripts/x.mjs

All paths are relative to skills/browser-cli/ within the project root.

Workflow Overview

1. Prepare — start daemon, open target site in a dedicated tab
2. Explore — identify page types; try markdown as a first pass for content pages
3. Discover Selectors — scan live DOM for testids, containers, field selectors; use network watch for API-heavy SPAs
4. Build & Validate — write extraction scripts incrementally, test on real data
5. Write Recipe Script — create the .mjs file with reusable functions
6. Write Guide — create the .md file with selector tables and boilerplate
7. Register — add to SKILL.md table

---

Step 1: Prepare

browser-cli start
browser-cli status          # confirm "Extension: connected"
browser-cli tab new '<site-url>' --group browser-cli
# Output: Tab <tabId>: <url> (group: browser-cli)

Save the tab ID — use --tab <tabId> for ALL subsequent commands to avoid disrupting user browsing.

browser-cli --tab <tabId> wait 3000
browser-cli --tab <tabId> snapshot -ic    # initial overview of interactive elements

Step 2: Explore Page Types

Identify the site's 2–5 key page types (home, search, detail, profile, etc.).

Check login requirements

browser-cli --tab <tabId> navigate '<page-url>'
browser-cli --tab <tabId> get url    # did it redirect to login?

Detect login state

browser-cli --tab <tabId> eval --stdin <<'EOF'
JSON.stringify({
  loginButton: !!document.querySelector('[data-testid*="login"], .login-btn, a[href*="login"]'),
  signupButton: !!document.querySelector('[data-testid*="signup"], .signup-btn'),
  avatar: !!document.querySelector('[data-testid*="avatar"], .user-avatar'),
})
EOF

Document which pages are public and which require login.

Quick content check

For content/article pages (news, docs, blogs), try markdown as a first pass before DOM discovery:

browser-cli --tab <tabId> navigate '<page-url>'
browser-cli --tab <tabId> markdown    # extracts readable content; may be enough without DOM scraping

If markdown returns the data you need, skip DOM selector discovery and go straight to Step 5.

Page type discovery

For each page type, start with snapshot -ic to get an overview, then drill into specific sections:

browser-cli --tab <tabId> navigate '<page-url>'
browser-cli --tab <tabId> wait 3000
browser-cli --tab <tabId> snapshot -ic              # interactive elements overview
browser-cli --tab <tabId> snapshot -c -d 3          # content structure (depth-limited)
browser-cli --tab <tabId> snapshot -ic -s @e5       # drill into a specific section

Step 3: Discover Selectors

Never guess selectors. For each page type:

3a. Scan data-testid attributes first (most stable)

browser-cli --tab <tabId> eval --stdin <<'EOF'
JSON.stringify([...new Set(
  [...document.querySelectorAll("[data-testid]")]
    .map(el => el.getAttribute("data-testid"))
)].sort())
EOF

3b. Find containers → inspect children → map fields

# Find container — try candidate selectors
browser-cli --tab <tabId> eval 'document.querySelectorAll("<selector>").length'

# Inspect structure of first 2 items
browser-cli --tab <tabId> eval --stdin <<'EOF'
JSON.stringify([...document.querySelectorAll("<container>")].slice(0, 2).map(el => ({
  tag: el.tagName,
  classes: el.className?.substring?.(0, 200),
  testId: el.getAttribute("data-testid"),
  text: el.innerText?.substring(0, 200),
  childTestIds: [...el.querySelectorAll("[data-testid]")].map(c => c.getAttribute("data-testid")),
  childTags: [...el.children].map(c => `${c.tagName}.${c.className?.split(' ')[0] || ''}`).join(', ')
})))
EOF

# Test individual field selectors
browser-cli --tab <tabId> eval 'document.querySelector("<container> <field-sel>")?.innerText'

3c. Monitor network traffic for API-heavy SPAs

For sites built on React/Vue/Angular that fetch data via XHR/fetch, use network watch to capture API traffic:

# Start monitoring API calls (non-blocking, writes to file)
browser-cli --tab <tabId> network watch '*api*' --timeout 10000 --body
# Navigate or interact to trigger requests, then stop
browser-cli --tab <tabId> network unwatch

If an endpoint returns clean JSON, extract via in-page fetch():

browser-cli --tab <tabId> eval --stdin <<'EOF'
(async () => {
  const resp = await fetch('/api/data', { credentials: 'include' });
  return await resp.json();
})()
EOF

Selector preference order

1. [data-testid="..."] — most stable, explicitly for testing
2. Custom element tag + attributes — Web Components store data in attributes (e.g., Reddit's shreddit-post with getAttribute("post-title")); no need to pierce shadow DOM
3. #id — unique IDs
4. [role="..."], [aria-label="..."] — semantic attributes
5. .semantic-class — human-readable class names (e.g., .hnuser, .score)
6. [class*="partial"] — CSS Module fuzzy match for hashed class names (e.g., [class*=_detail_] on Weibo); use when class names have random suffixes
7. Avoid: exact auto-generated classes (.css-1a2b3c), deep nesting

Step 4: Build & Validate Extraction Scripts

Build incrementally — start minimal, add fields one at a time:

# Minimal — verify container works
browser-cli --tab <tabId> eval --stdin <<'EOF'
JSON.stringify([...document.querySelectorAll("<container>")].slice(0, 3).map(el => ({
  text: el.querySelector("<title-sel>")?.innerText || ""
})))
EOF

# Add more fields after titles confirmed
browser-cli --tab <tabId> eval --stdin <<'EOF'
JSON.stringify([...document.querySelectorAll("<container>")].map((el, i) => ({
  index: i + 1,
  title: el.querySelector("<title-sel>")?.innerText || "",
  author: el.querySelector("<author-sel>")?.innerText || "",
  url: el.querySelector("a")?.href || "",
})).filter(r => r.title))
EOF

Validate thoroughly

• Test on 2–3 different queries/pages to ensure selectors generalize
• Check for empty fields, zero-value edge cases
• Filter out noise elements (ads, recommendations mixed into containers)
• Verify across different content types (image vs. text posts, etc.)

Test interactions

Anti-bot tip: If clicks or fills are silently rejected by the site, add --debugger to dispatch trusted CDP events (isTrusted=true). Example: browser-cli --tab <tabId> click '<sel>' --debugger. Chrome only; falls back to synthetic events on Firefox.

# Pagination — infinite scroll
browser-cli --tab <tabId> scroll down --amount 2000
browser-cli --tab <tabId> wait 1500

# Pagination — next page button
browser-cli --tab <tabId> click '<next-page-selector>'
browser-cli --tab <tabId> wait '<container>' --timeout 5000

# Filters
browser-cli --tab <tabId> click '<filter-toggle>'
browser-cli --tab <tabId> wait '<filter-panel>' --timeout 3000

# Detail navigation + back
browser-cli --tab <tabId> click '<item-link>'
browser-cli --tab <tabId> wait '<detail-container>' --timeout 5000
browser-cli --tab <tabId> back
browser-cli --tab <tabId> wait '<list-container>' --timeout 5000

Step 5: Write Recipe Script

Create skills/browser-cli/scripts/<short-name>.mjs with reusable functions.

Script conventions

• File header comment: purpose + note about browser.evaluate() auto-unwrap
• Each function: JSDoc with description and @requires precondition
• console.log: Log each step for debugging (navigating, waiting, extracting, result count)
• Default export: Full workflow function combining the individual steps
• Named exports: Individual functions for flexibility (--call <name>)

Template

// scripts/<name>.mjs — <Domain> recipe functions
// Each function: (browser, args?) => Promise<result>
// browser.evaluate() auto-unwraps { value } and JSON.parse, so expression
// that returns JSON.stringify(...) will yield a parsed JS object directly.

/** Detect login state → { loggedIn, ... }
 * @requires Current tab is on <domain> */
export async function detectLogin(browser) {
  console.log('Detecting login state...');
  const result = await browser.evaluate({
    expression: `JSON.stringify((() => {
      const loggedIn = !!document.querySelector('<logged-in-selector>');
      return { loggedIn };
    })())`,
  });
  console.log('Login state:', result.loggedIn ? 'logged in' : 'not logged in');
  return result;
}

/** Navigate to <page type>
 * @requires None */
export async function navigateTo(browser, { query } = {}) {
  const url = `https://<domain>/<path>?q=${encodeURIComponent(query)}`;
  console.log(`Navigating to: ${url}`);
  await browser.navigate({ url });
  await browser.wait({ selector: '<ready-selector>', timeout: 5000 });
  console.log('Page loaded');
}

/** Extract <items> → [{ index, title, url, ... }]
 * @requires Current page is a <page type> */
export async function extractItems(browser) {
  console.log('Extracting items...');
  const results = await browser.evaluate({
    expression: `JSON.stringify([...document.querySelectorAll("<container>")].map((el, i) => ({
      index: i + 1,
      title: el.querySelector("<title-sel>")?.innerText || "",
      url: el.querySelector("a")?.href || "",
    })).filter(r => r.title))`,
  });
  console.log(`Extracted ${results.length} items`);
  return results;
}

/** Full workflow: navigate and extract */
export default async function (browser, args) {
  await navigateTo(browser, { query: args?.query });
  return await extractItems(browser);
}

Advanced patterns

For complex sites, these patterns from existing scripts are worth adapting:

• Web Components with attribute-based data — some sites (e.g., Reddit) use custom elements where data is in element attributes rather than inner DOM. Extract via el.getAttribute("post-title") instead of el.querySelector(...)?.innerText. Shadow DOM children are accessed via slot: [...el.children].find(ch => ch.getAttribute('slot') === 'comment'). Reference: scripts/reddit.mjs → extractFeed / extractComments.
• CSS Module fuzzy matching — sites using CSS Modules have hashed class names (e.g., _detail_zsq3w). Use [class*=_detail_] partial match instead of exact class names. Reference: scripts/weibo.mjs selectors throughout.
• Virtual scroll accumulator — when items are removed from the DOM as you scroll (virtualized lists), inject window globals to track seen items across scroll batches. Reference: scripts/xhs.mjs → initScrollCollector / scrollAndCollect / getCollected.
• Network watch for API extraction — for sites with auth-bearing XHR URLs, use network watch to capture traffic, then re-fetch via in-page fetch(). More stable than DOM selectors for data-heavy SPAs. Reference: scripts/youtube.mjs → fetchTimedtext.
• contentEditable input — rich text editors don't respond to fill. Use document.execCommand('insertText') after focusing the element. Reference: scripts/xhs.mjs → postComment.
• Draft.js / contentEditable search — some inputs (e.g., Discord search) are contentEditable divs, not <input>. Use CDP key dispatch (press --debugger) for character-by-character input. Reference: scripts/discord.mjs → searchMessages.

Testing the script

Write to a temp file and test before committing:

TMP=$(mktemp /tmp/bcli-XXXX.mjs)
cat > "$TMP" <<'SCRIPTEOF'
// paste your script here
SCRIPTEOF
browser-cli --tab <tabId> script "$TMP" --list                       # verify exports
browser-cli --tab <tabId> script "$TMP" --call detectLogin           # test individual function
browser-cli --tab <tabId> script "$TMP" --call extractItems          # test extraction
browser-cli --tab <tabId> script "$TMP" -- --query "test"            # test full workflow

Step 6: Write the Guide

Create skills/browser-cli/references/sites/<domain>.md.

Language: Write in the site's primary user language (Chinese sites → Chinese).

Guide template

Every guide starts with the same header boilerplate, then has selector reference tables:

# <domain>

> One-line site description.

> **Tip**: To avoid disrupting user browsing, open a dedicated tab first:
>
> ```
> browser-cli tab new 'https://<domain>' --group browser-cli
> ```
>
> Then use `--tab <tabId>` for all subsequent commands.

> **Recipe scripts**: Common operations are encapsulated as reusable script functions. Read `scripts/<name>.mjs` source to see available recipe functions and their preconditions (`@requires`).
>
> ```bash
> # Call a specific function
> browser-cli --tab <tabId> script scripts/<name>.mjs --call extractItems
> # Pass arguments to a function
> browser-cli --tab <tabId> script scripts/<name>.mjs --call navigateTo -- --query "test"
> ```
>
> When the agent runs, replace `scripts/<name>.mjs` with the absolute path (derived from the SKILL.md directory).

> **Recipe debugging**: If a recipe function fails (e.g. selectors changed), copy the function from `scripts/<name>.mjs`, modify the selectors, and re-run via `script -` (stdin):
>
> ```bash
> browser-cli --tab <tabId> script - <<'EOF'
> export default async function(browser) {
>   // Copied from <name>.mjs extractItems, with modified selectors
>   return browser.evaluate({
>     expression: `JSON.stringify([...document.querySelectorAll("<container>")].slice(0,3).map(el => ({ title: el.querySelector("<title>")?.innerText || "" })))`
>   });
> }
> EOF
> ```
>
> You can also debug selectors step by step with `eval`: `browser-cli --tab <tabId> eval 'document.querySelectorAll("<container>").length'`
>
> See the selector tables below for reference.

## Selector Reference

### Login Detection

| State     | Selector | Notes |
| --------- | -------- | ----- |
| Logged in | `...`    | ...   |

### <Page Type 1> (e.g., Search Results)

**URL pattern**: `/<path>?<key params>`

| Element           | Selector | Notes |
| ----------------- | -------- | ----- |
| Results container | `...`    |       |
| Item title        | `...`    |       |
| Item link         | `...`    |       |

### <Page Type 2> (e.g., Detail Page)

**URL pattern**: `/<path>/<id>`

| Element | Selector | Notes |
| ------- | -------- | ----- |
| ...     | `...`    |       |

## Common Interactions (optional)

Include for complex sites with multi-step flows that agents will frequently need:

### <Action> (e.g., Search)

```bash
# Direct URL (most reliable)
browser-cli --tab <tabId> navigate 'https://<domain>/search?q=<query>'
browser-cli --tab <tabId> wait '<results-container>'

# Or via recipe
browser-cli --tab <tabId> script scripts/<name>.mjs --call <function> -- --query "test"
```

## Notes

- Gotcha 1
- Gotcha 2

Key points

• Selector tables, not inline scripts: The guide focuses on selector reference tables. Extraction logic lives in the .mjs recipe script.
• No duplicate extraction scripts: Don't copy extraction code from the .mjs into the .md. The guide points to the script; the script is the source of truth.
• Notes section: Document gotchas — CSP issues, auth requirements, rate limiting, selector stability, pagination behavior, locale differences.

Step 7: Register

Add to the "Site-Specific Guides" table in skills/browser-cli/SKILL.md:

| <domain> | [sites/<domain>.md](references/sites/<domain>.md) |

Quality Checklist

Before finishing, confirm:

• Every selector was discovered from the live DOM (not guessed)
• Recipe script tested with --call for each exported function
• Default export (full workflow) tested end-to-end
• Extraction returns populated data on 2–3 different pages/queries
• Zero-value edge cases handled (0 likes, 0 comments)
• Filter/sort interactions tested if documented
• Pagination tested (next page, scroll load)
• Login detection works (both logged-in and logged-out states)
• Auth or permission requirements noted in Notes section
• Guide registered in SKILL.md table
• Guide uses the standard header boilerplate (Tip + Recipe scripts + Recipe debugging)

Existing Guides (read one before writing)

Guide	Script	Best for learning
news.ycombinator.com.md	hn.mjs	Simple selectors, static HTML, comment tree
google.com.md	google.mjs	Search flow, multiple result types, time filter
x.com.md	x.mjs	data-testid selectors, login detection, SPA
xiaohongshu.com.md	xhs.mjs	Virtual scroll, contentEditable, SVG state
mail.google.com.md	gmail.mjs	CSP handling, inbox extraction
youtube.com.md	youtube.mjs	Player API, transcript/captions, SPA
reddit.com.md	reddit.mjs	Web Components, Shadow DOM, attribute-based extraction
weibo.com.md	weibo.mjs	CSS Module fuzzy match, Chinese guide, comment scroll
discord.com.md	discord.mjs	aria-label/role selectors, Draft.js input handling