upgrade-polars

# Upgrading the supported Polars version in cudf-polars This skill describes how to move the `polars` range that cudf-polars supports (for example `>=2.0,<2.1` to `>=2.1,<2.2`). It is distilled from the previous upgrade PRs, weighted toward the Polars 2.0 upgrade (rapidsai/cudf#24549). A bump is done when all of these pass locally: 1. The cudf-polars unit tests, with **100% coverage** (CI uses `--cov-fail-under=100`). 2. The upstream polars test suite with the injected GPU engine (`in-memory` and `spmd`). 3. The narwhals run for cudf-polars (`narwhals_cudf_polars_test_plugin`, the second block of `ci/test_narwhals.sh`). The cudf and cudf.pandas runs do not exercise polars. They only matter if Narwhals itself is bumped, which the `update-narwhals-testing` skill covers (it requires all three to pass). 4. `pre-commit` (including mypy and `ci/check_cudf_polars_ir.py`). Do the main work against the **newest** polars you support: coverage and the upstream polars tests only run on it in CI. Then run the cudf-polars unit tests on the **oldest** supported polars too (Step 4). CI's wheel job (`wheel-tests-cudf-polars`, `POLARS_VERSIONS=earliest` in `ci/test_python_cudf_polars.sh`) does exactly that, and a test, xfail or assertion changed for the new version often breaks there. ## Prerequisites Ask the user for: - a working GPU (check `nvidia-smi`). Build the environment yourself in Step 2 the way CI does; only ask the user for one if that is not possible; - the target Polars version (default: the newest release, `pip index versions polars`); - the target Narwhals version, only if narwhals must change. Ground rules: - One Polars minor version per PR. Do not bundle unrelated fixes; split them out (a reviewer asked for this in #24549). - Never leave a test failing and never hide wrong results behind an xfail. Prefer making the translator raise `NotImplementedError` (CPU fallback) over an xfail when the GPU result would be wrong. - Small root-caused fixes go in the upgrade PR. Larger work gets a drafted issue, linked from the xfail/skip reason or a `TODO`. Draft issues for the user to review before opening them. - Record which tests were run and their outcome as you go; the PR description needs it. - On an existing PR branch, sync with GitHub first (see "Resuming work on an open PR"). Never force-push. - Install the target version into the dev environment first, then confirm with `python -c "import polars; print(polars.__version__)"`. If a test environment variable or plugin is missing, say so instead of silently skipping. ## Practical notes - Run long suites in the background with output to a log file, and check them later rather than waiting inline. - The narwhals pin lives in `dependencies.yaml` (`depends_on_narwhals`); read it there instead of assuming a version. - **CI under-reports failures.** Every test job stops at its first failure (`-x`, or xdist "stopping after N failures"), so a red PR shows only a handful of tests. In the Polars 2.1 upgrade CI showed 3 unit and 4 upstream failures per job; the full local runs had 37 unit failures and 153 (in-memory) / 65 (spmd) upstream failures. Treat CI as a smoke signal and triage from full local runs (`--maxfail=300`, `--maxfail=100000`). Read CI logs with `gh pr checks <pr>` and `gh run view --job <job-id> --log-failed`. - **Shared GPUs.** Check `nvidia-smi` before every run and never disturb other people's work. Use one GPU per process (`CUDA_VISIBLE_DEVICES`) and keep concurrent jobs on different GPUs. The streaming engine (and the xdist plugin runs) abort with `No GPUs found on current NUMA node N` when the process runs on CPUs of the other NUMA node. Find the GPU's node with `nvidia-smi topo -m` and pin: `numactl --cpunodebind=<node> --membind=<node> python -m pytest ...`. The `in-memory` engine is unaffected. - `pytest-cov` can under-report: lines such as `raise NotImplementedError(...)` that a test provably executes may show as missed in a full run. Before adding a pragma or a test, confirm a line is really unreachable by tracing it: run the query under `sys.settrace` (or with `pytest <that test file> --cov`) and see whether the line executes. Only lines that are not reached by any query on the new polars need a test or `# pragma: no cover`. - Upstream tests marked `write_disk` are deselected unless you pass `-m ""` (CI passes `-m "not slow"`); remember this when running a single upstream test. - The narwhals repo keeps its package under `src/narwhals`. To test the pinned version without installing it, put `<clone>/src` (not the clone root) first on `PYTHONPATH`, and print `narwhals.__file__` to confirm which copy is imported. - **Parallel work.** Independent failure clusters can be handed to subagents. Give each its own `git worktree` and branch off the current PR commit, its own GPU, a list of exact test ids, and the rule "fix and test on your branch, or recommend an xfail with a drafted issue". Agents must not edit shared dicts like `inject_gpu_engine.py`; you merge their commits with `git cherry-pick` and resolve the (usually append-only) conflicts. Give each agent a private scratch directory; shared scratch scripts get overwritten. ## Resuming work on an open PR (keep local and GitHub in sync) The branch on GitHub can change while you are away: the user squashes or rebases, force-pushes, accepts a review suggestion, edits the PR in the web UI, or a reviewer pushes. Your local checkout does not know until you fetch. Run this check before you edit, commit or push anything on an existing PR, and again right before every push. 1. Fetch and compare. `<branch>` is the PR branch: ```bash git fetch origin <branch> git status -sb git log --oneline HEAD..origin/<branch> # commits on GitHub that you do not have (remote ahead) git log --oneline origin/<branch>..HEAD # commits you have that GitHub does not (local ahead) ``` 2. Act on what you find: - **Both empty:** in sync; continue. - **Only the remote is ahead:** `git pull --ff-only origin <branch>`, then read what arrived (`git log -p ORIG_HEAD..HEAD`) before you build on it. - **Only you are ahead:** push normally, with no force. - **Both are non-empty (diverged):** find out whether history was rewritten. Test whether the remote still contains the last commit you pushed (`git merge-base --is-ancestor <last-pushed-sha> origin/<branch>`; `git merge-base HEAD origin/<branch>` should print that sha). - **It does** (the user merged `main`, added commits, or accepted a suggestion): nothing was rewritten. Rebase your unpushed commits onto the remote with `git rebase -S origin/<branch>` (`-S` keeps them signed), tell the user what arrived, and read the incoming diff for files you also touch. Check that your last verification still applies: a merge of `main` can change code your tests cover. - **It does not** (a squash, rebase or force-push): stop and tell the user. Do not merge, and do not force-push on your own. With the user's agreement: first keep a copy (`git branch backup/<branch>-$(date +%F)`), then move only your new commits onto the remote with `git cherry-pick` (your older commits are already inside the squash), resolve conflicts toward the remote side's intent, and check the result with `git diff origin/<branch>`. 3. Never `git push --force`. Use `--force-with-lease` only when the user explicitly asks for a history rewrite (for example to squash), because it refuses to overwrite a remote that moved since your last fetch. 4. Re-read the PR before changing anything on it: `gh pr view <n> --json headRefOid,body,reviews,comments,reviewDecision`, the inline review comments (`gh api repos/<org>/<repo>/pulls/<n>/comments`) and CI (`gh pr checks <n>`). The user may have edited the description, so merge your change into the current body; do not overwrite it from an old local file. The same applies to issues (`gh issue view` before `gh issue edit`). 5. After pushing, confirm the remote head is your commit (`git ls-remote origin <branch>`). Workflows on a draft PR do not start on their own: the user has to comment `/ok to test <sha>` for the new head. 6. After a squash by the user, your old commit hashes are no longer in the PR. Refer to commits by subject, and fix any note that quoted a hash. State in the PR which commit the last full test run was on, so later edits can be judged against it. 7. A branch can be checked out in only one worktree, so use the worktree that has it. Do not use bare `git stash` (the stash is shared across worktrees); commit instead. 8. When you address a review comment, say what you changed. Replying to or resolving threads is the user's call unless they ask you to. ## Step 1. Survey what changed upstream Start by shallow-cloning Polars at the target tag into a temporary directory and keep it for the whole upgrade. It is the reference for this survey, for the exact logic behind a behaviour change, and for the upstream test suite in Step 5: ```bash git clone --depth 1 --branch py-X.Y.Z https://github.com/pola-rs/polars.git <tmp>/polars git -C <tmp>/polars fetch --depth 1 origin tag py-OLD # only needed to diff against the old version git -C <tmp>/polars diff py-OLD py-X.Y.Z -- crates/polars-python/src/lazyframe/visitor py-polars/src/polars/_plr.pyi ``` Read the Polars release notes for every version between the old and new pin (`gh api repos/pola-rs/polars/releases/tags/py-X.Y.Z -q .body`; they are long and mostly irrelevant to cudf-polars, so skim for the items below) and, when a note is unclear, read the code in the clone (the visitor in `crates/polars-python`, optimizer rules in `crates/polars-plan`). List: removed or renamed APIs, new IR nodes or `Function`/`BooleanFunction`/`StringFunction`/`StructFunction`/ `TemporalFunction` enum members, changed option payloads, changed error types/messages, changed dtype semantics, and new deprecation warnings (the test suite runs with `filterwarnings = error`). Keep the list; it drives steps 3 to 5. Also read the Polars upgrade guide (https://docs.pola.rs/releases/upgrade/), which lists the breaking changes and deprecations for each release. Check each item against cudf-polars; behaviour changes that the GitHub release notes only mention briefly are often described properly there. Beyond release notes, the most reliable signals are the tests themselves: `test_from_polars_all_names` for enum drift, and the diffs of `crates/polars-python/src/lazyframe/visit.rs` and `py-polars/src/polars/_plr.pyi` between the two tags (in the clone) for IR changes. Also run `git log` and grep the tree for stale guards that a merge may have reintroduced: `grep -rn "POLARS_VERSION_LT\|POLARS_LOWER_BOUND\|polars_compat" python ci dependencies.yaml`. ## Step 2. Bump the pins All the places that carry the range (edit the exact same string everywhere): | File | What | | --- | --- | | `dependencies.yaml` | `run_cudf_polars` list: `polars>=X,<Y` | | `dependencies.yaml` | `test_cudf_polars_compat`: the `polars_compat_version` list under `files:` and the matching `polars==X.*` entries under `dependencies:` (one block per supported minor, plus the fallback entry). CI scripts read this matrix, so they usually do not need edits | | `python/cudf_polars/pyproject.toml` | **generated** from `dependencies.yaml` by `rapids-dependency-file-generator` (the list carries a "generated" marker); do not hand-edit, confirm it changed after the generator run | | `conda/recipes/cudf-polars/recipe.yaml` | `polars >=X,<Y` | | `.pre-commit-config.yaml` | mypy hook `additional_dependencies` | | `conda/environments/all_cuda-*_arch-*.yaml` | **generated** by the same hook; run `pre-commit run --all-files rapids-dependency-file-generator` rather than hand-editing | | `python/cudf_polars/cudf_polars/utils/versions.py` | `POLARS_LOWER_BOUND` only if the minimum moves | Default for a new minor: **raise the upper bound only** (`>=2.0,<2.1` becomes `>=2.0,<2.2`) and append the new minor to the `test_cudf_polars_compat` matrix (`["2.0", "2.1"]`, plus a `polars==2.1.*` block next to the existing one). Do not drop the previous minor in the same PR. Policy for the lower bound: support roughly the last 5 or 6 minor versions, but cloud defaults (Colab, AzureML, Kaggle) can justify a lower minimum. Confirm the intended range with the user before editing, and ask before raising the minimum. When the upgrade adds support for a new **major** Polars version (for example 2.0), the lower bound moves too: raise it to the first minor of that major, update `POLARS_LOWER_BOUND`, and remove the guards, code and tests that only supported the older major (see below). Regenerate the generated files (the conda environment files and the `pyproject.toml` dependency list) with the env's own `pre-commit` (it is usually not on `PATH`): `export PATH=<env>/bin:$PATH; pre-commit run --all-files rapids-dependency-file-generator`. The first run "fails" because it modified files; that is expected. Run it again to confirm it is clean. It needs network access to initialise the hooks. When the minimum moves, delete every guard for versions below it (and the code, tests, `request` fixtures and xfail conditions that only existed for them). Do **not** reintroduce `POLARS_VERSION_LT_*` flags speculatively. Add one to `versions.py` only when the code must behave differently on two supported versions, and delete it when the lower bound passes it. Parenthesize conditional expressions (`x == (A if cond else B)`); an unparenthesized one was a real bug in an earlier upgrade. ### Remove run-list entries for dropped versions PR CI selects upstream tests from `ci/cudf_polars/gpu_test_run_list.json`, which has one entry per supported Polars version (`<version>|spmd|default`). Regenerating the list keeps entries for old versions, so when the lower bound moves, remove the entries below it in the same PR. After the `pyproject.toml` regeneration above, run from the repo root: ```bash python .agents/skills/update-cudf-polars-polars-tests-selection/prune-run-list.py ``` The script reads the minimum supported version from the `polars` requirement in `python/cudf_polars/pyproject.toml`, prints each entry it removes, and rewrites the list in place. Check that `git diff ci/cudf_polars/gpu_test_run_list.json` only removes entries for versions below the new minimum. The newest supported version also needs an entry, or PR CI fails before running any tests. Generate it with the `update-cudf-polars-polars-tests-selection` skill once the upstream suite passes (Step 5). ### Build the environment the way CI does Do this right after the pin and the generator run, so the environment matches what CI will use: 1. Edit the polars pin in `dependencies.yaml` (the `run_cudf_polars` list and the `test_cudf_polars_compat` matrix, see the table). 2. Run `rapids-dependency-file-generator` (the pre-commit hook above) so the generated files pick up the change. 3. Create an environment from `dependencies.yaml` or from a generated file. CI builds it with `rapids-dependency-file-generator --output conda --file-key <key> --matrix "cuda=<ver>;arch=<arch>;py=<ver>;dependencies=<latest|oldest>"` followed by `mamba env create -f` (`ci/test_python_common.sh`); the cudf-polars job uses the keys `test_python_other` and `test_cudf_polars_compat` (`ci/test_python_cudf_polars.sh`, which also reads the polars version from the matrix). For a local environment, `conda env create -n <name> -f conda/environments/all_cuda-<ver>_arch-<arch>.yaml` is the same set of packages. If the new polars release is not on conda-forge yet, `pip install polars==X.Y.Z` into the environment and run `pip check`. 4. Build or install cudf-polars and the related libraries into it: `pip install -e python/cudf_polars`, with pylibcudf and libcudf from the nightly channel unless you are changing C++. Confirm `python -c "import polars, cudf_polars; print(polars.__version__)"`. To reproduce a failing CI job in its own container instead, use the `reproduce-ci` skill. ## Step 3. Adapt the translator and expressions Run the code against the new polars and fix in this order. 1. **IR version gate.** `Translator.translate_ir` in `cudf_polars/dsl/translate.py` rejects IR versions it has not been validated against. The error message prints the version of the polars you are *adopting* (for example `(16, 0)`), which is a version you must now support, **not** the value to put in the gate. Once the code works against it, set the gate to the next minor, `>= (16, 1)`. Keep the `# pragma: no cover` on the raise. 2. **Enum drift.** `tests/dsl/test_serialization.py::test_from_polars_all_names` compares polars' function name enums with ours exactly. For each new member add `Name.X = auto()` (alphabetical) in `dsl/expressions/{boolean,string,struct,temporal}.py`. If it is unsupported, ensure it raises `NotImplementedError` so it falls back to CPU, and add a translation-raises test. Track unimplemented new features in a drafted issue. 3. **Changed IR/option payloads.** When a node gains or reshapes a field (for example `MergeSorted.key` becoming a list, a new `HConcat` slot, `Sort.slice` becoming a 3-tuple), update the translate function and the IR class. When an IR class signature changes, grep every constructor call (translator, `streaming/`, tests). `ci/check_cudf_polars_ir.py` validates `do_evaluate` signatures. 4. **New fallbacks.** If a new option cannot be honoured on GPU, raise `NotImplementedError` from the translator (or the node constructor) with a clear message. Add `assert_ir_translation_raises(q, engine, NotImplementedError)` tests; where the *upstream* test cannot pass on GPU, this is the preferred fix. 5. **Semantic changes.** Fix when small (with a unit test). If it needs libcudf work or a redesign, fall back to CPU and draft an issue. 6. **Pass the stream.** Pass `stream=df.stream` to every pylibcudf call you add or touch (reviewers sweep for it). Prefer `Column.astype` over hand-rolled cast checks, and drop `hasattr` guards for APIs that now always exist. 7. **Imports.** Use `from polars import _plr as plrs` for the private Rust bindings. mypy needs the overrides already in the root `pyproject.toml` for `polars._plr`. 8. **Polars constant-folds literals into scans.** In 2.1 `select(pl.lit(<struct/enum>))` becomes a `DataFrameScan`, so the translator never sees a `Literal` and a test that expects `NotImplementedError` from that query stops raising. Keep the raise test but build the query with `with_columns` (or `when/then`), which still reaches the `Literal`, and add a small `assert_gpu_result_equal` test for the folded `select` form. Check what a changed test now exercises (print `translator.errors`), not just that it passes.