j-crawl

name	j-crawl
description	Extract clean Markdown from web pages via the self-hosted Crawl4AI service at $CRAWL4AI_URL. Use when WebFetch returns junk on JS-rendered or anti-bot pages, or for whole-site ingestion. Keyless — never install a crawler, puppeteer, or hosted Firecrawl.

/j-crawl

Self-hosted Crawl4AI reachable at $CRAWL4AI_URL (already passed into every container — do not hardcode the host). Keyless: no API token today, no per-container secret. NEVER install a crawler, puppeteer, playwright-extra, or reach for hosted Firecrawl — the endpoint is already there.

When to use what

• Find pages → WebSearch (built-in). Crawl4AI is not a search engine.
• Ordinary page → text → WebFetch first. It's cheaper.
• JS-rendered / anti-bot / whole-site ingestion → Crawl4AI /md (below). Reach here only when WebFetch returns junk or JS-gated content.
• Clicks, forms, multi-step → playwright-cli (local browser), not Crawl4AI.

The /md call

curl -s -X POST "$CRAWL4AI_URL/md" \
  -H 'Content-Type: application/json' \
  -d '{"url": "https://example.com", "f": "fit"}'

• f: "fit" is the heuristic content filter (default; no LLM call). Omit for default.
• Omit c entirely. It must be a string if sent ("0"); boolean c: false → HTTP 422.
• Add -H "Authorization: Bearer $CRAWL4AI_TOKEN" only if a token is set (it isn't today).

The response (flat)

{"url": "...", "filter": "fit", "markdown": "# ...", "success": true}

• Markdown is the top-level markdown field, gated by success: true — NOT nested under result.markdown.fit_markdown.
• Derive the title from the first # heading.

Then synthesize from the returned Markdown.