crawler-discovery-workflow-codex

name

crawler-discovery-workflow-codex

description

Use when Codex is asked to build, debug, evaluate, or integrate a crawler/data-collection workflow for a new website, new data source, or new data type. This skill enforces a staged engineering process: compliance and scope check, data contract definition, site exploration, API/HTML/rendered/WAF classification, single-site POC scripting, data quality reporting, stabilization, reusable adapter extraction, dedup/review handling, and service/backend integration. It is not a "universal crawler" skill and should not promise that every site can be crawled.

Crawler Discovery Workflow

This skill makes new-site data collection repeatable and auditable. It does not guarantee that any website can be crawled. It forces a disciplined path from requirement to POC to quality report to reusable integration.

Non-Negotiable Workflow

Do not skip steps unless the user explicitly asks for a narrow diagnostic.

1. Check compliance and scope.

   • Identify target domain, data type, intended use, access level, robots/terms risk, login/cookie requirements, and whether the user has authorization.
   • Do not bypass paywalls, private areas, CAPTCHA, account restrictions, or explicit access controls unless the user confirms authorization.
   • See references/legal-and-ethics.md.

2. Define the data contract.

   • List required fields, optional fields, unique key candidates, source URL fields, freshness requirements, and output destination.
   • Missing text should be represented consistently; missing arrays should be []; missing numbers should be 0 when the target pipeline expects non-null values.
   • See references/data-contract.md.

3. Explore before coding.

   • Prefer JSON API, then SSR HTML, then embedded JSON, then browser-rendered extraction.
   • Record pagination, detail page dependency, rate limits, WAF symptoms, cookies, headers, encoding, SSL issues, and whether JavaScript is required.
   • See references/site-analysis.md and references/api-first-playbook.md.

4. Build a single-site POC first.

   • One website, one data type, one script.
   • Save raw samples where useful, normalized JSON output, and a crawl result report.
   • Report count, valid count, duplicate count, field fill rate, failed pages/URLs, warnings, and whether snapshots were used.
   • See references/poc-script-patterns.md and templates/.

5. Evaluate data quality before calling it successful.

   • Crawler success is not "script exits 0".
   • Crawler success means data quantity is plausible, required fields are filled at explainable rates, failures are traceable, and duplicate handling is auditable.
   • See references/data-quality.md.

6. Stabilize only after the POC works.

   • Add retry, timeout, rate limit, user-agent/header strategy, checkpoint/resume, structured logging, and explicit fallback behavior.
   • Do not silently fall back to stale snapshots. If a snapshot is used, expose snapshot_used: true, the reason, and the snapshot source.
   • See references/anti-bot-and-fallback.md and references/observability.md.

7. Convert to a reusable adapter only after evidence.

   • POC before abstraction.
   • Two successful site-specific POCs before extracting a shared framework.
   • Three similar adapters before introducing base classes.
   • Keep site-specific URL patterns, selectors, API parameters, and fallback logic isolated.

8. Validate dedup and review flow before production writes.

   • Dedup by source ID and canonical URL first.
   • Mark cross-source suspected duplicates; do not silently drop uncertain records.
   • Generate a crawl report before writing production tables.
   • See references/dedup-review.md.

9. Integrate as a staged pipeline.

   • POC script -> result JSON -> service adapter -> callback/job status -> intermediate review table -> reviewer edit/approve/reject -> production import.
   • Do not bypass review tables unless the user explicitly requests a one-off import script.
   • See references/integration.md.

Tool Selection

Use the least complex method that returns reliable data.

1. JSON API: fastest and most stable when public/authorized.
2. Raw HTTP or Scrapling FetcherSession for SSR HTML.
3. Embedded JSON in HTML or script tags.
4. Browser rendering with Playwright/Scrapling DynamicSession only when data is client-rendered and no API is usable.
5. Stop and report when access controls, CAPTCHA, private auth, or legal constraints block collection.

Scrapling is a preferred option, not a mandatory dependency. It is useful for adaptive selectors, fetchers, spiders, TLS impersonation, and anti-bot ergonomics, but API clients, urllib/httpx, and Playwright can be the right choice.

Required Artifacts

For a new data source, produce or update:

• Site report: use templates/site_report.md.
• POC script: use templates/poc_crawler.py.
• Field mapping: use templates/field_mapping.yaml.
• Crawl result report: use templates/crawl_result_report.md.
• Normalized result JSON.
• Notes on failures and fallback strategy.

Hard Rules

• Do not start by building a generic framework.
• Do not rely on fixed user-provided item counts; discover site counts or crawl until pagination ends.
• Do not hide live crawl failures behind local snapshots.
• Do not call data "deduped" unless the compared tables/keys are stated.
• Do not claim full success without a data quality report.
• Do not overwrite existing crawler POC files or result snapshots without checking their contents.
• After code changes, run the smallest end-to-end check that covers probe/explore, crawl, normalization, dedup/review insertion, and status/progress.

Reference Map

• Compliance: references/legal-and-ethics.md
• Full process: references/workflow.md
• Site exploration: references/site-analysis.md
• API-first collection: references/api-first-playbook.md
• Scrapling usage: references/scrapling-playbook.md
• POC script patterns: references/poc-script-patterns.md
• Anti-bot/fallback: references/anti-bot-and-fallback.md
• Data contract: references/data-contract.md
• Data quality report: references/data-quality.md
• Dedup/review: references/dedup-review.md
• Observability: references/observability.md
• Integration: references/integration.md