Menu
Add a Yahoo Sports source — capture + catalog in sdv-internal-refs/yahoo (HTML/JSON scrape).
Installer avec Codex ou Claude Copiez ce prompt, collez-le dans Codex, Claude ou un autre assistant, puis laissez-le vérifier la page du skill et l'installer pour vous.
Basé sur la classification professionnelle SOC
| name | sdv-add-yahoo-source |
| description | Add a Yahoo Sports source — capture + catalog in sdv-internal-refs/yahoo (HTML/JSON scrape). |
| disable-model-invocation | true |
Adds a new Yahoo Sports data source to the sdv-internal-refs/yahoo catalog. Yahoo Sports is a scrape (no clean public API). Data is embedded as JSON in HTML pages or available from partial XHR endpoints observable via browser DevTools. Treat all Yahoo captures as brittle with a capture date.
| Property | Value |
|---|---|
| Base URL | https://sports.yahoo.com |
| Auth | None (session cookie for some personalized pages; not required for public data) |
| Data delivery | Embedded JSON in HTML (root.App.main = {...} or __PRELOADED_STATE__) OR partial XHR JSON |
| Catalog | c:\Users\saiem\Documents\sdv-internal-refs\yahoo\ |
| Body captures | c:\Users\saiem\Documents\sdv-internal-refs\yahoo\inputs\sample_bodies\ |
| Fragility | HIGH — Yahoo restructures pages across seasons; always date-stamp captures |
Open https://sports.yahoo.com/<sport>/<page> in a browser with DevTools open.
Method A — Page source search:
curl -s -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36" \
"https://sports.yahoo.com/<sport>/<page>" \
| grep -o 'root\.App\.main = {.*' | head -c 500
The blob is typically root.App.main = {...}; or window.__PRELOADED_STATE__ = {...}; in a <script> tag.
Method B — XHR intercept (browser DevTools → Network → Filter: Fetch/XHR):
Navigate to the page, look for requests returning application/json. Common XHR patterns:
https://api-secure.sports.yahoo.com/v1/editorial/...https://sports.yahoo.com/_td/<sport>/...https://query1.finance.yahoo.com/... (finance crossover for sports betting lines)Document the full URL + any required headers (User-Agent, Referer, Cookie if applicable).
import re, json, httpx, pathlib
url = "https://sports.yahoo.com/<sport>/<page>"
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"}
resp = httpx.get(url, headers=headers, follow_redirects=True)
html = resp.text
# Try root.App.main pattern
m = re.search(r'root\.App\.main\s*=\s*(\{.+?\});\s*\n', html, re.DOTALL)
if not m:
# Try __PRELOADED_STATE__ pattern
m = re.search(r'__PRELOADED_STATE__\s*=\s*(\{.+?\});\s*\n', html, re.DOTALL)
blob = json.loads(m.group(1))
out = pathlib.Path("yahoo/inputs/sample_bodies/<sport>/<page>.json")
out.parent.mkdir(parents=True, exist_ok=True)
out.write_text(json.dumps(blob, indent=2))
print(f"Saved {out} — top-level keys: {list(blob.keys())[:10]}")
The embedded blob is large (often 300KB–2MB). Navigate the key path:
import json, pathlib
blob = json.loads(pathlib.Path("yahoo/inputs/sample_bodies/<sport>/<page>.json").read_text())
# Common paths:
# blob["context"]["dispatcher"]["stores"]["GamesStore"]["games"]
# blob["context"]["dispatcher"]["stores"]["TeamStore"]["teams"]
# blob["context"]["dispatcher"]["stores"]["StandingsStore"]["standings"]
# Walk it:
def walk_keys(d, depth=0):
if depth > 3 or not isinstance(d, dict):
return
for k, v in d.items():
print(" " * depth + k, type(v).__name__,
f"({len(v)} items)" if isinstance(v, (list, dict)) else "")
walk_keys(v, depth + 1)
walk_keys(blob)
Document the key path (e.g. context.dispatcher.stores.GamesStore.games) in the returns doc.
Create yahoo/catalogs/<sport>_<page>_returns.md:
# Yahoo Sports — <Sport> <Page> Returns
**Source type:** HTML embedded JSON (scrape)
**Captured:** YYYY-MM-DD
**URL:** `https://sports.yahoo.com/<sport>/<page>`
**Blob key:** `root.App.main`
**Data path:** `context.dispatcher.stores.<Store>.items`
| Field | Type | Notes |
|---|---|---|
| id | string | |
...
## Extraction selector
```python
blob["context"]["dispatcher"]["stores"]["<Store>"]["items"]
_td/<sport>/ endpoints
---
## Step 5 — Extend the Yahoo catalog
Open `yahoo/yahoo_catalog.yaml` and add:
```yaml
- sport: <sport>
page: <page>
source_type: html_embedded_json
blob_key: root.App.main
data_path: context.dispatcher.stores.<Store>.items
capture_date: YYYY-MM-DD
fragility: high
returns_doc: catalogs/<sport>_<page>_returns.md
_td/<sport>/ pattern is observable today but has changed before. Treat XHR captures as lower-reliability than embedded JSON.time.sleep(1) between requests in any bulk capture script.fragility: high in the catalog. This informs downstream decisions about whether to maintain a Yahoo-backed wrapper vs. deprecating it.Use when porting R logic into the Python SportsDataverse package (sdv-py) — e.g. translating an nflfastR / cfbfastR / cfbscrapR / baseballr / hoopR R function, reconciling a fix from the pandas `0.36-live` branch into polars `main`, or re-implementing a modeling recipe (EP/WP/QBR/CPOE) from R. Enforces a parity-test-first workflow (golden fixture from the R output → failing parity test → port → green) and the polars-1x + ID-dtype + no-lookaround conventions. Also covers R numeric fidelity when outputs must match R bit-for-bit (R>=4.0 fround, 80-bit long-double sum, non-na.rm null-poisoning, dplyr group order), stale-fork sibling packages (wbigballR-style), and oracle integrity (a browser/rvest-captured oracle can be corrupt — fix the bug, don't match it). Invoke for "port this R function to polars", "port from nflfastR/bigballR", "reconcile 0.36-live into main", "translate this dplyr/np.select to polars", "re-implement the R model recipe in Python", or when a parity test fails on a handful of rounding-boundar
Use when shipping a change in sportsdataverse-py (sdv-py) — opening, pushing, or merging a PR. Runs the steps in the correct order (regenerate codegen docs, update changelog/docs/tutorials, lint, full pytest, commit + verify it landed, push, triage bot reviews CodeRabbit/Sourcery/Copilot immediately — in parallel with CI, not after it — wait for CI green, confirm merge, write a session note) and never cleans up a branch before the merge is confirmed. Invoke for "ship this", "open/merge the PR", "land this change", or any end-of-change release flow.
Use when executing a model-spine implementation plan (the ClaudeCowork specs/plans backlog — prediction stacks, ratings engines, projection/impact models) or building any oracle-gated analytics module in an SDV repo. Covers the full loop — isolated worktree + baseline, Phase-0 oracle harness (metrics/constants/leakage split/fixtures), per-task TDD with verified commits, oracle gates with the never-lower rule, league-shim parity, mypy/codegen close-out, reviewer pass, and the session restart prompt. Invoke for "implement T<x>", "start the <sport> model spine", "continue the prediction stack", or any multi-phase model build.
Use when capturing an external oracle/validation corpus into committed test fixtures — Torvik/KenPom ratings, ESPN BPI/predictor/odds, market closing lines, MoneyPuck, published RAPM/EPM, or any third-party reference a model gate will assert against. Covers column contracts, Utf8-id discipline, the team/player name-crosswalk recipe (contracting normalizer + candidate keys + alias table + match-rate reporting), rate-limit-aware sampling, and the mandatory provenance README. Invoke for "capture the oracle corpus", "commit the <source> fixtures", or any Task-0-style fixture capture in a model plan.
Use when porting Python logic from the SportsDataverse Python package (sdv-py) into a SportsDataverse R package — cfbfastR, hoopR, wehoop, baseballr, fastRhockey, softballR, etc. Enforces a parity-test-first workflow (golden fixture from the Python output → failing testthat test → port → green) plus the tidyverse/data.table idiom map and R package conventions (roxygen2 docs, snake_case, tibble returns, pkgdown reference coverage). Invoke for "port this to cfbfastR", "translate this polars logic to dplyr", "re-implement the sdv-py parser in R", or "bring this Python fix back to the R package".
Add a CBS Sports (NAPI) league/resource — capture + catalog in sdv-internal-refs/cbs.