upload-method

# Process & Upload a Benchmarked Method to TabArena This skill drives the maintainer workflow that turns a benchmark run's **already-present** raw `results.pkl` files into cached, hosted (r2), and registered TabArena artifacts. There is **no download and no auto-generation** — the raw data is assumed to be on disk already (e.g. you unzipped a submission, or it's a fresh run's `output/<run>/data` dir). The authoritative prose for this flow is **AGENTS.md → "Processing & uploading method artifacts (maintainers)"** and the two scripts' module docstrings (`scripts/run_process_method.py`, `scripts/run_upload_results.py`). This skill operationalizes it: **Claude executes the flow itself by default**, and makes the small deterministic code edits along the way. ## What this skill delivers 1. **A plan, stated first**: which methods, which run dir, which suite, and that the flow ends in a real r2 upload — tell the maintainer before executing (no need to wait for approval unless they asked to review something first). 2. **Claude-run execution**, in order per method: - `inspect` — read the raw data, print inferred fields + a metadata diff. - `process` — build + cache `metadata.yaml` + `processed/` + `results/` locally. - `upload` (dry-run, then `--no-dry-run`) — push the cached artifacts to r2. 3. **Code edits**: - The model's `info.py` `MethodMetadata` — fill in the manual upload fields, fix any raw-data mismatches the inspect diff surfaces. - The arena collection registration in `methods.py` so the method appears in the benchmark. - The run's entry in `packages/tabflow_slurm/BENCHMARK_LOG.md`, if the launch stage never wrote one. 4. **A fallback command sheet** only for steps the environment can't run. Execution requirements (check before starting; hand off the affected step if missing): - a venv with **`tabarena[benchmark]`** installed (look under `~/.venvs/tabarena_*`; run scripts with its python from the repo root), - **R2 credentials in the environment** for the real upload (`printenv | grep -o '^R2_[A-Z_]*'` should show `R2_ACCOUNT_ID` / `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY`; never pass them as flags). Processing and uploads are long-running: run them **in the background** (per-method), watch with a monitor (per-method DONE/FAILED lines + a stall watchdog), and for multi-method batches pipeline the uploads — start each method's dry-run + real upload as soon as its processing finishes. ## Step 0: Gather inputs Parse `$ARGUMENTS`. Collect (ask only for what's missing or ambiguous): | Input | Example | Notes | |---|---|---| | `run_data_dir` | `.../output/benchmark_chimeraboost_16062026/data` | Dir of raw `results.pkl` files (searched recursively). Point at the run's `data/`. | | `model` | `chimeraboost` | The `packages/tabarena/src/tabarena/models/<model>/` folder — or `systems/<system>/` when the method is a whole pipeline (AutoML framework, agent, hosted API). Usually inferable from the run/data dir name; confirm. | | `suite` | `tabarena-2026-06-30` | The dated run/suite id. **Must differ from `method`** (see Step 3). Default to `tabarena-<run-date>`; ask if unclear. | | `arena` | `tabarena` | Which arena collection to register in (default `tabarena`; e.g. `beyondarena` has its own). | | `verified` | `False` until signed off | Whether the results are verified. Default keep `False`; flip to `True` only when the maintainer confirms (Step 3). | ## Step 1: Locate the method's `MethodMetadata` Read `packages/tabarena/src/tabarena/models/<model>/info.py` (or `systems/<system>/info.py`) and note the **exact** variable name of its `MethodMetadata` (e.g. `chimeraboost_method_metadata`, `tabfm_plus_method_metadata`). Both scripts take it as a dotted reference: ``` tabarena.models.<model>.info:<varname> tabarena.systems.<system>.info:<varname> ``` - **`info.py` exists** (the normal case — the model was added via the `add-model` skill): it already carries the raw-inferable fields (`ag_key`, `config_default`, `can_hpo`, `is_bag`, `compute`, `method_type`). `validation_protocol` is inferable too but was not authored before the run: a new run records it, and `--process` fails until `info.py` declares the inferred key. Claude fills it together with the *manual* upload fields in Step 3. - **No `info.py`** (a raw external submission): the method isn't integrated yet. Run `inspect` (Step 2) to get the copy-paste `MethodMetadata.<type>(...)` snippet, then author `models/<model>/info.py` from it (use the **`add-model`** skill if the model also needs a wrapper, or **`add-system`** if it is a whole pipeline). Processing **requires** an explicit committed `MethodMetadata` — it refuses to guess. - **The method is a system**: the inspector cannot tell a system from any other baseline, since the runner records both identically. Use `MethodMetadata.system(...)` rather than `.baseline(...)`, and set `tags` — see the `add-system` skill. Getting this wrong types the method as a model on the leaderboard and puts it in the models-only entrant pool, where it does not belong. ## Step 2: Inspect the raw data (Claude runs; confirms inference) ```bash <venv>/bin/python scripts/run_process_method.py <run_data_dir> ``` This prints the fields inferred from the raw data (`method_type`, `ag_key`, `compute`, `config_default`, `can_hpo`, `is_bag`, `validation_protocol` with a fold histogram, task/problem-type/metric coverage), a suggested `MethodMetadata` snippet, and — when an `info.py` metadata is passed — an **inferred-vs-provided diff**. Use it to confirm `info.py` matches the raw data before processing. Two gotchas it catches: - **`config_default` is compared post-rename**: configs are renamed to the method's prefix during processing, so a `config_default` authored with the raw prefix won't match. The snippet/diff shows the post-rename value — use that. The most common real mismatch is the infix, not the prefix: the TabArena-v0.1 bundle names the first config `<Method>_c1_default_BAG_L1` (the `_default` is the preprocessing pipeline name, appended for HPO and default-only models alike), while `info.py` files authored before a run, and the BeyondArena bundle, use `<Method>_c1_BAG_L1`. An `info.py` written by `add-model` before the run therefore usually needs this one fix. A single-config method (`can_hpo=False`) can drop the field instead: an undeclared `config_default` is not checked (the diff marks it `not declared`) and `--process` records the lone config in the cached `metadata.yaml`. - **Only `error` rows gate processing**: the `method` row differs whenever the raw `ag_name` carries the `TA-` prefix (warn-only) and `model_key` is never checked (shown for context). A `NO` on either is expected and needs no edit; a `NO` on `config_default`, `ag_key`, `compute`, `can_hpo`, `is_bag` or `method_type` does. - **`method != suite`**: `process` fails if they're equal (suite defaults to method when unset). - **`validation_protocol` must be declared**: a run made after the protocol record existed infers a key (`8x1` for TabArena, `system` for a system, the long BeyondArena key); `--process` fails while `info.py` leaves it `None` or declares another value. Legacy raw artifacts infer `None` and pass. Add `--expect-validation-protocol tabarena` (or `beyondarena`) to also assert that the run used the arena's official protocol; a `holdout:`/`outer`/custom key there means the run cannot be uploaded as an official result. KNN's `8x1 requested; 1 child via use_child_oof` histogram line is expected. **BeyondArena runs**: the task directories of a BeyondArena run are named `<dataset>-<hash>`, not OpenML task ids, so `EndToEnd.from_path_raw` cannot infer the task metadata from them and filters every result out (`No results to process`). Pass the committed collection, `task_metadata=load_beyond_task_metadata_collection("BeyondArena")` (from `tabarena.evaluation.beyond_metadata`), which `scripts/run_process_method.py` does not expose yet: process through a small driver that calls `EndToEnd.from_path_raw` with the `RawMethod`'s resolved name / model key / suite (see the `beyondarena_tfms_22092026` driver in `tmp_scripts/`). Register the method in `contexts/beyondarena/methods.py` through the model's `ModelDescriptor` (`<x>_descriptor.method_metadata(...)` with the file's `_common_fm_kwargs` / `_common_bag_kwargs`), adding a descriptor to the model's `info.py` when it only has `MethodMetadata.config(...)` entries; the BeyondArena bundle names configs `<Method>_c1_BAG_L1` (no `_default` infix) and `method` follows the raw `ag_name` (`TA-...` for most wrappers, `Causilo` for Causilo). **Multi-method run dirs**: if the run's `data/` holds several methods' config dirs side by side, `run_process_method.py` can't split them — write a small gitignored driver in `tmp_scripts/` that discovers the `results.pkl` paths once, splits them by top-level config-dir prefix, and calls `_infer_from_raw` / `verify_method_metadata` / `process_raw(file_paths=...)` per method (see the `multi-method-run-upload` pattern; a `--c1-only` pass over just the `_c1_` dirs makes `config_default` inferable even for HPO methods with hundreds of configs). ## Step 3: Edit `info.py` (Claude does now) Fill in the **manual** fields the upload requires. These are not inferable from raw data, so the maintainer normally hand-edits them — Claude does it now. Read the file first, then `Edit`: | Field | Set to | Why | |---|---|---| | `suite` | the dated run id, e.g. `"tabarena-2026-06-30"` | **Required, must differ from `method`.** Equal pair fails `process`'s `method != suite` check (suite defaults to method when unset). | | `cache_type` | `"r2"` | **Required for upload** — the upload script is r2-only; `"local"`/`None` has no remote store. | | `cache_kwargs` | `{"bucket": "tabarena", "prefix": "cache"}` | The r2 location (`bucket` + `prefix`). Required when `cache_type="r2"`. | | `date` | `"YYYY-MM-DD"` | The run date. Validated as a real calendar date. | | `verified` | `False` (until signed off), then `True` | Manual trust flag. Keep `False` until the results are verified; flip to `True` once they are (typically the final step). | | `method_class` / `tags` | systems only | `MethodMetadata.system(...)` sets `method_class`; `tags` (`with-llm`, `closed-source-api`) decide which entrant pools the system competes in. Neither is inferable from raw data. A system with several configs (`<Name>_c1_default`, `_c2_default`, ...) is hosted as one `MethodMetadata` per config, like the AutoGluon presets in `systems/autogluon/info.py`. For a `closed-source-api` system `compute` is the provider's hardware: the raw results record the CPU client node, and the inspect step shows that mismatch as a warning, not an error. | | `validation_protocol` | the key the inspect step infers, e.g. `"8x1"` | The inner validation protocol the results record. `--process` requires it for a new run; `run_upload_results.py` refuses an upload whose processed artifact records a key `info.py` does not declare. `MethodMetadata.system(...)` defaults it to `"system"`. | Leave the other raw-inferable fields (`ag_key`, `config_default`, `can_hpo`, `is_bag`, `compute`, `method_type`) **as they are** — they came from `add-model`. Only change one if Step 2's diff shows a genuine mismatch (and then to the inferred value; for a single-config method, deleting a mismatched `config_default` is equally valid). Example — making `chimeraboost` upload-ready (the upload fields added; compare `nori`'s `info.py`, which is already in this shape): ```python chimeraboost_method_metadata = MethodMetadata.config( method="ChimeraBoost", suite="tabarena-2026-06-30", # added: distinct dated suite ag_key="CHIMERA", config_default="ChimeraBoost_c1_default_BAG_L1", # HPO method: post-rename, `_default` = the run's pipeline compute="cpu", is_bag=False, date="2026-06-15", reference_url="https://github.com/bbstats/chimeraboost", display_name="ChimeraBoost", verified=False, # flip to True once verified cache_type="r2", # added cache_kwargs={"bucket": "tabarena", "prefix": "cache"}, # added ) ``` ## Step 4: Execute process → upload (Claude runs, in the background) Run these with the resolved dotted reference, per method — background + monitor for the slow ones. The same block doubles as the fallback command sheet if a step must be handed to the maintainer. Using `chimeraboost` as the example: ```bash # 1. Inspect (confirm inferred fields match info.py) — already done in Step 2 # 2. (info.py edited: suite + cache_type/cache_kwargs + date, plus any inspect-diff fixes) # 3. Process: build + cache metadata.yaml + processed/ + results/ locally <venv>/bin/python scripts/run_process_method.py <run_data_dir> \ --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata --process \ --expect-validation-protocol tabarena # the run must record TabArena's official protocol # 4. Upload dry-run: verifies every part exists locally and prints what/where (no creds needed) <venv>/bin/python scripts/run_upload_results.py \ --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata # 5. Real upload (R2 creds via env, NEVER flags): <venv>/bin/python scripts/run_upload_results.py \ --method-metadata tabarena.models.chimeraboost.info:chimeraboost_method_metadata --no-dry-run ``` After the real upload, verify on the bucket (ListObjects via boto3 against the R2 endpoint): each method should have the full object set under `cache/artifacts/<suite>/methods/<Method>/` (`metadata.yaml`, `processed.zip`, `processed/configs_hyperparameters.json`, `raw.zip`, `results/*.parquet`). The real upload also records the MD5 of each uploaded results table in `packages/tabarena/src/tabarena/models/_artifacts/results_checksums.json`. Commit that diff together with the `info.py` / `methods.py` edits: `download="auto"` loaders re-download a cached table that differs from its committed checksum, and `tests/tabarena/tools/test_results_checksums.py` fails in CI when a registered method has no entry or an entry differs from the hosted object. For a re-upload under an existing suite the diff changes the old entries, which is what makes other machines replace their cached copies. `python -m tabarena.tools.results_checksums --check` compares