| name | benchmark-model |
| description | Scaffold the setup + eval scripts to run one model on the TabArena benchmark cluster. Use this skill whenever a maintainer wants to benchmark an already-integrated model — e.g. "benchmark TabM", "run Nori on the cluster", "create a setup/eval script for DenseLight", "I want to launch <model> on TabArena and evaluate it". Generates a single `tmp_scripts/run_<model>.py` with `setup` and `eval` subcommands that share one benchmark_name + paths (so they cannot drift), auto-filling GPU/CPU, eval subsets, the install reminder, foundation-model prefetch from the model's registry `info.py`, and — mandatory for GPU models — the VRAM budget (`fake_memory_for_estimates`; asks the user when the partition's VRAM is not inferable). Complements `add-model` (integrate a model) and `upload-method` (publish its results). |
| argument-hint | <ModelRegistryName> [<benchmark_name>] [<n_configs>] |
| user-invocable | true |
Benchmark a Model on TabArena
This skill scaffolds the throwaway experiment scripts a maintainer runs to launch one already-integrated model on the SLURM cluster and then evaluate it. It replaces the old two-file pattern (run_setup_<model>.py + run_eval_<model>.py) with one tmp_scripts/run_<model>.py exposing setup and eval subcommands.
Why one file: setup and eval must use the same benchmark_name and PathSetup (workspace + python_path) — the eval side reads the raw results from path_setup.get_output_path(benchmark_name)/data that setup wrote. Splitting them across two files makes that identity drift. Here it is defined once at the top (BENCHMARK_NAME, WORKSPACE, PYTHON_PATH, MODEL) and read by both.
The generated file is a verbatim, self-contained plan (no hidden helpers) so its setup() body can be copied into packages/tabflow_slurm/BENCHMARK_LOG.md, and it lands in gitignored tmp_scripts/.
The golden template is references/run_benchmark_template.py — copy it and fill the <...> / # EDIT markers. Do not hand-write the structure.
Step 0: Gather inputs
Parse $ARGUMENTS. Collect (ask only for what's missing or a genuine judgment call):
| Input | Example | Default / source |
|---|
MODEL | "TabM" | required — the registry name (see Step 1) |
BENCHMARK_NAME | "tabm_26062026" | <model-lowercased>_<DDMMYYYY> using today's date |
NUM_CONFIGS | 0, 25, "all" | 0 if the model has no HPO search space, else ask (foundation models → 0; tunable models → "all" or a capped int under compute limits) |
| task scope | full / lite / regression / dataset names | TaskSubset() — the full task set (all splits); TaskSubset(subset="lite") for a quick first-split smoke run |
gpu_partition | "gpurtxpro6000spotinteractive" | that partition (the current default GPU node); ask if a bigger GPU is needed |
fake_memory_for_estimates | 96 | required for every GPU model — the partition's VRAM in GB (see Step 1a); ask the user if you cannot determine the VRAM from context |
| per-run overrides | time_limit | none unless the model/partition needs them |
WORKSPACE and PYTHON_PATH default to the values in the template (the shared cluster workspace + the tabarena_18062026 venv). Keep them unless the user names a different venv/workspace.
Step 1: Introspect the model registry (auto-fill the mechanical bits)
Given MODEL, read the model's contribution under packages/tabarena/src/tabarena/models/<key>/ and derive:
| Derived value | Where to read it | Drives |
|---|
compute ("cpu"/"gpu") | info.py → ModelDescriptor(compute=...) / MethodMetadata.compute | resources={"num_gpus": 1} + name="gpu" for GPU; drop the override + name="cpu" for CPU |
| problem types | model.py → the _supported_problem_types class attribute (a subset of ["binary","multiclass","regression"]; absent = all types). Read it via model_cls.supported_problem_types() | the eval subsets: all-types → [[], ["binary"], ["multiclass"], ["regression"]] ([] = the full set / overall leaderboard); regression-only (e.g. Nori) → [["regression"]] and scope setup with task_subset=TaskSubset(subset="regression") |
| HPO search space | info.py → search_space (a gen_<key> generator); empty/absent ⇒ no HPO | default NUM_CONFIGS (foundation models with no real search space → 0) |
| pip extra | info.py → ModelInfo(pip_extra=...) | the "install into the run venv" reminder in the docstring + Step 4 |
| weights prefetch | info.py → ModelInfo(prefetch_weights=...) (not None ⇒ foundation model) | a docstring note that the checkpoint is fetched from HF by the registry before the fits (no per-script action) |
| static memory estimate | model.py → whether the class implements _estimate_memory_usage_static (AutoGluon 1.6 derives can_estimate_memory_usage_static from its presence) | whether fake_memory_for_estimates can actually cap fold-parallelism (Step 1a caveat) |
Prefer reading these files over importing the model (no optional deps needed). If the venv already has the model installed, you may confirm quickly with:
<PYTHON_PATH> -c "from tabarena.models.utils import get_model_info_from_name as g; i=g('<MODEL>'); print(i.method_metadata.compute, i.pip_extra, i.prefetch_weights, i.model_cls.supported_problem_types())"
Step 1a: GPU model ⇒ always set fake_memory_for_estimates
Every GPU job must carry "fake_memory_for_estimates": <VRAM_GB> in its ModelJob.resources.
AutoGluon budgets parallel bagging folds by comparing the model's memory estimate against the
reported memory limit (node RAM by default) and splits the GPU only fractionally — VRAM is never
accounted. On a RAM-rich node, 8 folds co-schedule on one card and OOM it (this killed every
APSFailure fit of the first tabm_26062026 TabM run: ~12 GiB VRAM × 8 folds on a 95 GiB card).
Reporting the VRAM as the budget makes the same scheduling arithmetic cap folds by VRAM; RAM-wise
it only makes the budget more conservative, which is safe on the VRAM<RAM nodes we use.
- Determine
<VRAM_GB> from context: the chosen gpu_partition (gpurtxpro6000spotinteractive
→ RTX PRO 6000 → 96), the user's request, or the node notes in
packages/tabflow_slurm/src/tabflow_slurm/setup/resources.py (BeyondArenaResourcesSetup:
40/80/96 GB nodes). If the partition's VRAM cannot be determined from context, ask the user —
do not guess.
- CPU models: never set it (their estimate must be compared against real RAM).
- Caveat — the cap works through the model's estimate: a model that does not implement
_estimate_memory_usage_static (e.g. TabFM, TabSwift) has no static estimate, so AutoGluon falls
back to a small data-size estimate and fold-parallelism stays at 8 regardless. Still set the value, but tell the
maintainer to sanity-check per-fold VRAM × 8 (or pin num_folds_parallel via
ag_args_ensemble) before launching.
Step 2: Generate tmp_scripts/run_<model>.py
Copy references/run_benchmark_template.py to tmp_scripts/run_<model-key>.py and fill every marker: the module docstring notes, BENCHMARK_NAME, MODEL, NUM_CONFIGS, the ModelJob resources (GPU: num_gpus plus the mandatory fake_memory_for_estimates from Step 1a; any time_limit), task_subset, gpu_partition, and the eval subsets. The template defaults to the full task set (TaskSubset()) and emits both PDF and PNG figures (figure_file_type=("pdf", "png")); narrow task_subset and subsets to lite for a quick smoke run. Remove the guidance comments that don't apply so the result reads like the existing per-model scripts. Keep it importable and lint-clean (from __future__ import annotations, 120-col).
For a CPU model: drop resources={"num_gpus": 1}, set name="cpu", and use scheduler_setup=GCPSlurmSetup(cpu_partition=...). For a local, no-SLURM run, swap GCPSlurmSetup for LocalSequentialSetup(continue_on_error=True) and python_path=sys.executable (see packages/tabflow_slurm/experiments/run_tabarena_v0pt1_local.py).
Step 3: Report the workflow + reminders
Give the maintainer the two commands and the pre-req:
# 1. one-time: install the model's extra into the run venv (if pip_extra is non-empty)
<PYTHON_PATH> -m pip install <pip_extra...>
# 2. launch on the cluster
<PYTHON_PATH> tmp_scripts/run_<model>.py setup
# 3. after the run completes, evaluate
<PYTHON_PATH> tmp_scripts/run_<model>.py eval
Then offer to record the run in packages/tabflow_slurm/BENCHMARK_LOG.md (newest-first entry, current git SHA, the verbatim setup() body, and the why in the notes) — that log is committed even though the script is not.
Notes
- The scripts target TabArena-v0.1 (
TabArenaContext + TabArenaV0pt1ExperimentBundle + TabArenaEvalConfig). BeyondArena is a different eval shape (BeyondArenaContext, BeyondArenaEvalConfig, BenchmarkRun comparisons) — for that, adapt packages/tabflow_slurm/experiments/run_beyondarena.py instead.
benchmark_name is a cache key: reuse it unchanged across reruns/eval; change it only for a genuinely new run.ModelJob(name=...) groups jobs by hardware — models with the same name share one sbatch command. Give GPU and CPU models different names to split them.
