| name | polars-migration |
| description | Use when migrating a PySpark job or function in this repo (jobs/) to Polars (fargate/), or when asked to "migrate to polars", "convert this PySpark function", "port this job to fargate". Walks the repo's migration checklist so nothing from the PR template gets missed. |
Migrating a PySpark job/function to Polars
This repo is mid-migration from PySpark (jobs/) to Polars (fargate/). When migrating, follow this checklist — it mirrors .github/PULL_REQUEST_TEMPLATE/polars_migration_template.md.
- Check for existing utils before writing new ones. Search the narrowest scope first: stage
utils/ → project utils/ → repo-wide projects/utils/. Only add a new util if nothing suitable exists at any of those scopes.
- Confirm correct placement for any new/moved util, using the scope table:
| Used by | Location |
|---|
| One job/dataset only | projects/<project>/<stage>/utils/*.py |
| Multiple jobs/datasets in a project | projects/<project>/utils/*.py |
| Multiple projects | projects/utils/utils.py |
- Use
LazyFrame for test data, not eager DataFrame.
- Add or port unit tests in pytest style — see the
pytest-pattern skill for the repo's dataclass+parametrize convention.
- Add Google-style docstrings, including non-obvious performance notes (why something stays lazy, why a
.collect() happens where it does).
- You have free rein to improve the function's name, docstring, and test names/structure as part of the conversion — don't feel bound to preserve old PySpark-era naming just for continuity. This is scoped to the function(s) actually being migrated, not an excuse to rename untouched neighbours.
- Tag the old PySpark function with a comment pointing at its replacement:
- Follow the S3/Athena naming convention when saving output — see CLAUDE.md's "Dataset naming in S3 / Athena" section.
- Update
CHANGELOG.md — see CLAUDE.md's "Changelog" section for the format.
- Remind the user of the PR-process steps that aren't yours to do: request AI code review, move the Trello ticket to the PR column.
Scope discipline: only migrate what's being asked. Don't mass-rewrite untouched PySpark files just because you're nearby, and don't refactor beyond what the migration needs (see the repo's general no-premature-abstraction guidance).
For scale/memory concerns specific to the new Polars code (laziness, .over() vs joins, streaming-engine coverage), see CLAUDE.md directly, or the polars-streaming-check / over-vs-join skills if the migrated logic touches those areas.
