| name | data-ingestion-and-crawling |
| description | Use when building or extending the project's web search, crawling, scraping, parsing, or source-normalization systems for sports, fantasy, and prediction workflows, especially when reliability, provenance, freshness, and site-compliance matter across development and production runtimes. |
Data Ingestion And Crawling
Use this skill when the task is to fetch, parse, normalize, or monitor external information sources.
Goals
- Maximize signal quality, not raw page volume.
- Preserve provenance from raw fetch through normalized record.
- Keep ingestion modular so connectors can be swapped or disabled.
- Prefer resilient pipelines over brittle one-off scrapers.
Source priority
- Official APIs and feeds
- Official websites and competition pages
- Reputable primary reporting
- Community sources and social signals
Connector workflow
- Define the source type and access method.
- Fetch raw artifacts with timestamps and identifiers.
- Parse into structured fields.
- Normalize entities into shared schemas.
- Score freshness and extraction confidence.
- Store both raw and normalized outputs.
- Record errors and parser drift for maintenance.
Minimum connector output
{
"source_name": "",
"source_type": "",
"fetched_at": "",
"canonical_url": "",
"entity_refs": [],
"raw_artifact_ref": "",
"parsed_facts": [],
"freshness_score": 0,
"parser_confidence": 0
}
Reliability rules
- Prefer scheduled polling plus event-triggered refreshes near deadlines.
- Build parsers that fail loudly when selectors or structures drift.
- Cache raw responses so extraction changes can be replayed.
- Deduplicate near-identical items before they reach agents.
Compliance rules
- Respect published terms, robots guidance, rate limits, and legal constraints.
- Prefer licensed, public, or permissioned access over adversarial scraping.
- Do not design anti-bot evasion as a default strategy.
Escalation points
Escalate when:
- the source has unstable structure,
- official data is unavailable,
- the same fact appears differently across sources,
- the connector may create legal, operational, or maintenance risk.
