| name | sdv-port-r-to-python |
| description | Use when porting R logic into the Python SportsDataverse package (sdv-py) — e.g. translating an nflfastR / cfbfastR / cfbscrapR / baseballr / hoopR R function, reconciling a fix from the pandas `0.36-live` branch into polars `main`, or re-implementing a modeling recipe (EP/WP/QBR/CPOE) from R. Enforces a parity-test-first workflow (golden fixture from the R output → failing parity test → port → green) and the polars-1x + ID-dtype + no-lookaround conventions. Also covers R numeric fidelity when outputs must match R bit-for-bit (R>=4.0 fround, 80-bit long-double sum, non-na.rm null-poisoning, dplyr group order), stale-fork sibling packages (wbigballR-style), and oracle integrity (a browser/rvest-captured oracle can be corrupt — fix the bug, don't match it). Invoke for "port this R function to polars", "port from nflfastR/bigballR", "reconcile 0.36-live into main", "translate this dplyr/np.select to polars", "re-implement the R model recipe in Python", or when a parity test fails on a handful of rounding-boundary cells. |
Port R → Python (polars), parity-test-first
The failure this prevents is the recurring one: a port looks right, lands, and the
divergence only surfaces downstream (a model rebuild, a join, a season compile) as a
wrong/empty result — then a fix → fail → revert cycle. The fix is to make the
R↔Python divergence a failing test before the port, not a surprise after it.
Create one todo per numbered step below.
1. Pin the canonical R source
Identify the exact upstream R you are porting and read it directly — don't port from
memory of how it "should" work. Canonical sources in the workspace:
nflverse-dev/nflfastR/R/*.R (e.g. helper_add_ep_wp.R, helper_add_fixed_drives.R,
helper_add_series_data.R) — the EP/WP/series/drives recipes.cfbfastR-dev/cfbfastR/R/*.R and the 0.36-live branch of sdv-py (pandas-flavored
CFB pbp fixes not yet in polars main; reconciliation notes live in dev/).baseball-dev/baseballr, hoopR-dev/hoopR, wehoop-dev/wehoop, hockey-dev/fastRhockey.
Quote the specific function + line range you are porting in the parity test docstring so
the provenance is durable.
If the source has a sibling package for another league (bigballR/wbigballR,
hoopR/wehoop), diff them before you trust either. The sibling is often a stale fork,
not an independent implementation: it inherits the parent's assumptions and silently
mis-handles its own league. (Real incident: wbigballR applies bigballR's two-halves
clock math to quarter-format WBB pages, so regulation reads as double-overtime.) Port ONE
parameterized core + thin per-league bindings, and treat a fork-inherited bug as a
deliberate fix — documented in-module, with the fixed behavior proven by an
independent invariant, never by matching the oracle's bug.
2. Capture a golden fixture (the R output)
Run the R function on a small, representative input and persist its output as the parity
oracle — CSV/parquet under tests/fixtures/ (or dev/ if it's a one-off reconciliation).
Capture the columns you will assert on plus the join keys. Note the R version + inputs
in a sibling README.md or the fixture header, matching the existing fixture provenance
convention.
The capture path can corrupt the oracle — verify it before you trust it. An oracle is
only as good as how it was captured, and R's HTML/browser readers mangle real payloads:
rvest::html_table (and any chromote/rendered-DOM path) expands a nested table's cells
into the parent row — so a column can hold a completely different stat than its name
claims — and a rendered DataTable re-sorts rows, so oracle row order ≠ source row order.
Prefer a static, offline capture over the exact bytes the Python test will parse
(XML::readHTMLTable, or the R package's own use_file=TRUE path against the committed
fixture) so both sides see identical input.
When a column is provably wrong-by-construction, do not port the bug to match it.
Partition the oracle columns and say so in the test docstring:
- strict — asserted cell-for-cell,
- promoted — empirically verified to agree despite being suspect (check before assuming
divergence; a constant offset often cancels inside the algorithm),
- invariant-covered — the oracle is wrong here; assert independent invariants instead
(bounds, monotonicity, totals) and prove the correction with a check the buggy version
could not pass (e.g. a 100% join-match rate that only holds under the fixed clock math).
If the R can't be run locally, hand-derive a few rows from the R logic by inspection and
mark them # derived-by-inspection — still better than no oracle.
3. Write the failing parity test FIRST
Before porting, add a test that loads the golden fixture and asserts the (not-yet-written)
Python output matches. Use correlation/closeness thresholds for float model columns and
exact equality for categorical/id columns. The nflfastR port's accepted bar is the
template: ep 0.996, epa 0.994, wp 0.997, vegas_wp 0.998 on the model domain; a
lower wpa ≈0.89 is an SNR ceiling, not a bug (it's exact when fed nflverse's own wp).
State the threshold and why in the test. Run it; confirm it fails for the right reason.
4. Translate idioms — pandas/dplyr/np → polars 1.x
| R / pandas idiom | polars 1.x |
|---|
dplyr::mutate(x = ...) / df.assign(...) | df.with_columns((...).alias("x")) |
dplyr::case_when(...) / np.select(conds, choices, default) | pl.when(c1).then(v1).when(c2).then(v2).otherwise(default) |
dplyr::group_by(g) %>% summarise(...) / df.groupby(...) | df.group_by("g").agg(...) |
dplyr::lag(x) / dplyr::lead(x) / df.shift() | pl.col("x").shift(1) / .shift(-1) — always .over("game_id") to avoid cross-game leak when frames are concatenated |
cumsum(x) | pl.col("x").cum_sum() |
ifelse(cond, a, b) | pl.when(cond).then(a).otherwise(b) |
df.loc[mask, col] = v | with_columns(pl.when(mask).then(v).otherwise(pl.col(col)).alias(col)) |
nrow() / dplyr::n() | pl.len() |
df[order(x), ] | df.sort("x") |
tidyr::pivot_wider | df.pivot(...) |
stringr::str_extract(s, re) | pl.col("s").str.extract(re, group) |
Hard rules (from sdv-py CLAUDE.md — these are the high-frequency port bugs):
- polars 1.x surface only. No
groupby / with_row_count / apply / pl.count /
cumsum / set_at_idx / how="outer" / str.strip. If you wrote a 0.18-era call,
it's a bug (the step-6 review pass catches these).
- No regex lookaround. Rust/polars regex rejects
(?=)/(?!)/(?<=)/(?<!). To
stop a capture at a stopword, use the inline case toggle (?i)prefix(?-i: NAMES).
- ID dtype discipline. Pick one canonical dtype per id at the boundary and keep it.
Never
cast(Utf8) a float-origin id (123.0 ≠ "123"); cast the raw integer. Assert
left.schema[key] == right.schema[key] before any join.
- Explicit boolean masks.
pl.col("c") == True, not bare pl.col("c").
- Float64 model outputs. Models emit float32; cast public columns explicitly so a
pl.Series(numpy_f32) can't silently downcast.
R numeric fidelity (when the port must match R bit-for-bit)
Signature: the parity test passes on 99% of cells and fails on a handful — almost all of
them .xx5 rounding boundaries, or a group sum that's off in the last ulp. That is not a
logic bug; it is R's arithmetic differing from Python's. Four sources, all real:
round() is not scale-and-round. R ≥4.0 (src/nmath/fround.c) takes the two
back-converted doubles floor(x·10^d)/10^d and ceil(x·10^d)/10^d, picks whichever is
nearer to x, and breaks an exact tie to the even scaled digit. So round(0.475,2)=0.48
but round(22.755,2)=22.75 — no single polars round mode (half_away_from_zero
or half_to_even) reproduces both, and neither does any decimal-repr rule. Port it as
an explicit _fround(x, digits) UDF (map_elements(..., return_dtype=pl.Float64)) and
fuzz-verify a boundary grid against local Rscript before trusting it.sum() accumulates in 80-bit long double. A plain float64 fold diverges at boundary
values (e.g. 42.765) and then flips the rounded output. Use math.fsum for any group sum
that feeds a rounded/reported column.- Non-
na.rm aggregations NULL-POISON. In R, sum(x) / max(x) without
na.rm=TRUE return NA if any element is NA — but polars aggregations skip nulls
and happily return a number. When the R call omits na.rm, add an explicit
any-null-poisons guard; don't let polars invent a value where R produced NA.
- dplyr group order = C-locale byte sort, NA groups last. If row order is part of the
contract,
.sort(keys, nulls_last=True) — and beware sorting a Utf8 id where R sorted a
numeric one (differs the moment ids have mixed digit-width).
These only matter for outputs that must match R exactly (rounded stats, published tables).
Don't reach for them on a model column governed by a correlation threshold.
5. Respect single-owner invariants
Some logic must live in exactly one place — don't re-add it inline during a port:
- EPA/WPA derivation lives only in
nfl/ep_wp.py (calculate_epa / calculate_wpa).
Construction modules emit a frame; ep_wp applies the models. Don't re-add EPA/WPA in a
pbp constructor.
- Player-name/id extraction for CFB lives in
cfb_play_participants.py; extend it
rather than adding new regex in cfb_pbp.__add_player_cols.
6. Go green, then run the gate
Implement until the parity test passes. Then run the fast inner loop:
uv run ruff format <changed.py> && uv run ruff check <changed.py>
uv run mypy
uv run pytest tests/<sport>/ tests/test_id_conventions.py -q
New modules must be fully typed and appended to the [tool.mypy] files ratchet. Then
dispatch the port-parity-reviewer agent on the ported module(s) — it is the
comprehensive post-port audit (ID-join dtypes, regex divergence, 1-based indexing,
NA/null/NaN drift, silent recycling, golden-master-test adequacy, module-name shadowing)
and hands polars currency to polars-1x-reviewer. Fix its MUST-FIX findings before
escalating to /sdv-preflight (scoped sweep) then /sdv-ship (full gate).
7. Record the reconciliation
If this was a 0.36-live → main port, note the function + commit in dev/ so the
function-by-function reconciliation map stays current. Don't merge 0.36-live wholesale —
it's pandas-flavored and would undo the polars migration; port semantic fixes by translation.
Fan-out mode (multi-module ports)
For a package-scale port (many independent R modules), parallelize steps 1–6 with
one implementer subagent per module instead of porting serially:
- Cap concurrency at ~6–8 — more trips a server-side rate limit. If completion
notifications look throttled, the on-disk outputs are ground truth, not the
notifications.
- Per-module gate: a module is DONE only when its golden-master parity test
(steps 2–3) passes — an agent's "done" report without a green parity test is
not done. Each agent's brief carries: the pinned R source path, the fixture
contract, the idiom hard-rules, and its report-file path.
- Ledger checkpoint: track per-module status in the SDD ledger, namespaced
under
.superpowers/sdd/<plan-slug>/ (flat task-N-* names clobber across
plans). On any interruption (rate limit, compaction, session end) resume from
the ledger + git log — never re-dispatch a module the ledger marks complete.
- Review wave: after each module goes green (not at the very end), dispatch
port-parity-reviewer on it — findings are cheapest before dependent modules
stack on top.
See also
port-parity-reviewer (agent) — the comprehensive post-port audit; dispatch it in step 6./sdv-build-data — for the heavy sweep that re-runs a ported pipeline over a corpus in a -data repo.sdv-port-python-to-r — the reverse direction (sdv-py → the SDV R packages).- sdv-py
CLAUDE.md "Polars version" + "ID column types" sections are the authoritative idiom/dtype reference.
