adapt-tabarena

# Adapt TabArena for a New Domain Benchmark This skill is for a **different repository** building its own benchmark on top of TabArena, not for contributing to this repo. If the user wants to add a model, system, or feature to TabArena itself, use `add-model` / `add-system` instead. The concrete reference implementation this skill distills is **RamanBench** (`github.com/ml-lab-htw/RamanBench`), which migrated its model/metrics/splitting layer onto `tabarena`/`bencheval` for its v1 release, replacing hand-rolled patterns that were "inspired by" TabArena with the real thing. Read that repo's `src/raman_bench/` for a worked example alongside this skill. Every module path below was checked against this checkout. Both packages are pre-1.0 and the API still moves (several paths cited by earlier drafts of this skill no longer exist), so `grep` a path before writing it into the downstream repo, and read the docstring of anything you depend on. ## What you reuse and what you write | Layer | Reuse from `tabarena` / `bencheval` | You write | |---|---|---| | Datasets and outer splits | `UserTask`, `TaskMetadataCollection`, `TaskMetadataSource`, `SubsetPredicate` | dataset loaders, the split construction, a committed metadata CSV | | Models and search spaces | the model registry, `ConfigGenerator`, every model's `hpo.py` | domain models registered via `register_model_info` | | Fitting protocol | the experiment bundles, `AGModelBagExperiment`, the AutoGluon wrappers, the `ValidationProtocol` the context asserts | a bundle subclass carrying your defaults, a context declaring your protocol | | Preprocessing | `TabArenaModelAgnosticPreprocessing`, `build_feature_generator`, model-specific hyperparameter injection | a domain feature generator or a model mixin | | Running | `AbstractArenaContext.build_and_run_jobs`, `ExperimentBatchRunner`, `JobBatch`, `tabflow_slurm` | a cluster profile | | Leaderboard math | `bencheval.evaluator.BenchmarkEvaluator`, reached through `compare` | nothing, or one `LeaderboardMetric` | | Publishing | `MethodMetadata` artifact tiers, `format_leaderboard`, the interactive HTML explorers | a `methods.py` roster and a place to host results | The canonical downstream loop, from `examples/benchmarking/run_quickstart_tabarena_custom_datasets.py`: ```python from tabarena.benchmark.experiment import TabArenaExperimentBundle from tabarena.benchmark.task.metadata import TaskMetadataCollection from tabarena.contexts import AbstractArenaContext tasks = [make_user_task(ds) for ds in my_datasets] # Step 2 collection = TaskMetadataCollection.from_user_tasks(tasks) # your suite experiments = TabArenaExperimentBundle( # Step 5 models=[("LightGBM", 0), ("RandomForest", 0), (my_gen, 10)], n_random_configs=50, preprocessing_pipelines=["default"], ).build_experiments(time_limit=3600) context = AbstractArenaContext(task_metadata=collection, methods=[]) # no TabArena baselines context.build_and_run_jobs(experiments, expname=results_dir, user_tasks=tasks, debug_mode=True) leaderboard = context.compare(output_dir=eval_dir) # Step 9 ``` Everything below explains the decisions hidden in those ten lines. ## Step 0: Scope the domain benchmark Ask (or infer from context) what the downstream benchmark actually needs: | Question | Why it matters | |---|---| | Full model zoo, or only leaderboard math over results you already have? | `tabarena` (models, runner, splitting) vs. `bencheval` alone (Elo, win-rates, ranks, improvability from a results DataFrame; no `tabarena` dependency). | | Do datasets have replicate or group structure (several rows per specimen, patient, sample)? | Both the outer splits (Step 3) and the inner bagging folds (Step 4) must be group-aware, and the inner ones need the `data-foundry` extra. | | Do datasets have temporal structure? | Outer splits must be forward-in-time; TabArena's inner protocol handles `time_on` but not `time_on` together with `group_on`. | | Do features arrive as thousands of numeric columns (spectra, expression arrays, embeddings)? | Foundation models carry feature and row caps; `ModelConstraints` (Step 5) skips incompatible pairs instead of failing them, and dimensionality reduction becomes a preprocessing decision (Step 7). | | Do the domain's metrics differ from `roc_auc` / `log_loss` / `rmse`? | Any metric must be registered with AutoGluon by name before a task can reference it (Step 2). | | Are there domain-specific models or whole pipelines? | Decides between a registered model (Step 6) and an `ExternalSystemModel`. | | Will you host results so others can compare against them without rerunning? | Decides whether you need a `methods.py` roster, `MethodMetadata` storage config, and your own arena context (Step 10). | | Cluster? PyPI release? | Steps 8 and 11. | ## Step 1: Depend on the right package(s) `bencheval` is standalone and light (numpy, pandas, scipy, scikit-learn). It computes leaderboards from a results DataFrame you already have. Pull it in alone if the domain benchmark keeps its own models and splitting and only wants TabArena-grade leaderboard math. Its `__init__` is empty, so import from submodules: `from bencheval.evaluator import BenchmarkEvaluator`. `tabarena` adds the model registry, config generators and search spaces, the task and experiment runner, the contexts, plotting, and the artifact tiers. It depends on `bencheval` and on an AutoGluon pre-release. Extras that matter downstream: `[benchmark]` for model fitting (the core model set plus `plot`, `text`, `preprocessing`, `data-foundry`), `[data-foundry]` on its own if you only need grouped inner splits (see Step 4), `[plot]` for the figures `compare` renders, and one extra per extended model (`[tabm]`, `[realmlp]`, `[tabpfn]`, ...). The registry's `pip_extra` on each `ModelInfo` names the extra a model needs. Both packages publish PyPI pre-releases at one shared version, and every `tabarena` release pins `bencheval==<same version>`. Install with `pip install --pre tabarena` (or `uv pip install --prerelease=allow`), and pin the pair together in the downstream `pyproject.toml`. See Step 11 before choosing between the PyPI release and a git URL. ## Step 2: Turn each dataset into a `UserTask` `tabarena.benchmark.task.UserTask` is the local, OpenML-free task type. Its `create_task` computes the task's `TabArenaTaskMetadata` (problem type, sizes, dtype flags, per-split statistics) and `save_task` pickles dataset, splits and metadata to one file. Reference implementations: `examples/benchmarking/run_quickstart_tabarena_custom_datasets.py` (plain DataFrames), `tabarena/benchmark/task/data_foundry/adapter.py::convert_curated_container_to_user_task` (the converter BeyondArena uses, a complete "third-party dataset object to `UserTask`" example), and `tests/tabarena/benchmark/task/test_user_task.py`. ```python from sklearn.model_selection import RepeatedStratifiedKFold from tabarena.benchmark.task import UserTask from tabarena.benchmark.task.metadata import GroupLabelTypes from tabarena.benchmark.task.user_task import from_sklearn_splits_to_user_task_splits n_folds, n_repeats = 3, 10 cv = RepeatedStratifiedKFold(n_splits=n_folds, n_repeats=n_repeats, random_state=0) splits = from_sklearn_splits_to_user_task_splits(cv.split(X, y), n_splits=n_folds) task = UserTask(task_name="ramanbench/bacteria-id", task_cache_path=task_cache_dir) wrapper = task.create_task( dataset=df, target_feature="species", problem_type="classification", splits=splits, eval_metric="log_loss", group_on="specimen_id", group_labels=GroupLabelTypes.PER_GROUP, ) task.save_task(wrapper) task.load().validate_metadata() # recomputes from disk and raises on any diverging field ``` The data contract `create_task` enforces: - `dataset` is one DataFrame that includes the target column, has a default `RangeIndex`, and has resolved dtypes. Numeric, `category`, `string` and datetime columns are supported; `object` columns raise. Categorical columns must already be `category`, not strings. - `splits` is `{repeat: {fold: (train_indices, test_indices)}}` with positional Python `int` lists. Numpy integers fail validation, so call `.tolist()` (the sklearn helper does). Train and test must not overlap, test indices must not overlap across folds of one repeat, and every repeat must have the same number of folds. A single holdout is `{0: {0: (train, test)}}`. - `task_name` must be unique on the machine: the cache file, the numeric task id and the results `dataset` key (a slug plus a hash of the name) all derive from it. - `eval_metric` must be an AutoGluon metric name. Defaults come from `tabarena.benchmark.task.metrics.DEFAULT_EVAL_METRIC_BY_PROBLEM_TYPE` (`roc_auc`, `log_loss`, `rmse`). To use a domain metric, register it once with `autogluon.core.metrics.make_scorer` and insert it into `autogluon.core.metrics.METRICS[<problem_type>]`, exactly as `tabarena/metrics/custom_metrics.py` does for the AMEX metric. Put the registration in a module every worker imports (your package `__init__` or the module that defines your tasks), otherwise a Ray worker resolving the metric by name will not find it. - `group_on`, `group_labels`, `time_on`, `group_time_on`, `stratify_on`, `split_time_horizon` and its unit are recorded on the task. They do not build the outer splits (you supply those); they drive the inner validation protocol (Step 4) and the preprocessing that wants group or time columns (Step 7). `GroupLabelTypes.PER_GROUP` means every row of a group shares the label (one specimen, many spectra); `PER_SAMPLE` means the label varies within a group (one patient, many visits). `time_on` together with `group_on` raises `NotImplementedError` downstream, and `split_time_horizon` is carried and logged but no split logic reads it yet. Where tasks live: without `task_cache_path` a `UserTask` is saved under the OpenML cache root in `tabarena_tasks/`, which `CacheConfig(openml=...)` relocates (Step 8). A task saved there has a path-free `task_id_str`, so `TaskMetadataCollection.from_user_tasks(tasks)` is runnable as-is. A task saved to a custom `task_cache_path` should be handed to the runner explicitly via `build_and_run_jobs(..., user_tasks=tasks)`, as the example does. `UserTask.from_task_id_str` reconstructs a task from the id string, which is what lets a compute node load a task it never created. ## Step 3: Outer splits: repeated k-fold, adaptive repeats, groups, time TabArena does not ship its own splitter. You build the outer splits with scikit-learn (or Data Foundry) and hand them to `create_task`. What to reproduce is the protocol: - Repeated k-fold, not a single holdout. TabArena v0.1 uses 3 outer folds and a repeat count that depends on the training-set size: 10 repeats below 2,500 training rows, 1 repeat above 250,000, otherwise 3. The policy is `_get_n_repeats` in `tabarena/benchmark/task/metadata/fetch_metadata.py` (a private function; port the three lines rather than importing it) and the constants are restated in `examples/advanced/run_get_tabarena_datasets_from_openml.py`. Verify a ported policy against `load_curated_task_metadata()` from the same module, not just against the docstring. - Grouped data has no repeated splitter in scikit-learn. Loop `StratifiedGroupKFold` / `GroupKFold` with `shuffle=True` and a different `random_state` per repeat, then feed the flat iterator to `from_sklearn_splits_to_user_task_splits`. Data Foundry's `get_recommended_grouped_splits` (in `data_foundry.curation_recommendations`) is the splitter TabArena itself uses for grouped inner folds, so using it for the outer splits keeps the two layers consistent. - Temporal data needs forward-in-time outer splits (train strictly before test). TabArena's inner helper `split_time_index_into_intervals` in `tabarena/benchmark/exec_models/autogluon_utils.py` bins a time column into row-balanced contiguous intervals; reuse it to define the outer test window, then hold out the latest interval. - Audit every dataset for group structure before the first real run. A dataset wrongly treated as row-independent leaks train information into test and inflates every model's score. RamanBench caught this late, on datasets that had already shipped without it. Decide the repeat count from data instead of guessing. `compare(output_dir=..., compute_fold_similarity=True, fold_similarity_kwargs={"target_reliability": 0.8})` writes `fold_similarity.csv` with a `folds_needed_for_stability@0.8` column per dataset (a Spearman-Brown extrapolation over the per-split rankings, from `BenchmarkEvaluator.rank_datasets_by_fold_similarity`). Take `min(folds_needed, num_folds)` per dataset, commit the resulting `(dataset, split)` table, and expose it as a `"core"` subset predicate with `tasks_in_frame` (Step 10). That is exactly how BeyondArena's `core` subset was derived; the recipe is `examples/!experimental/run_generate_beyondarena_core_subset.py`. The same mixin offers `dataset_representativeness` and `jitter_all_datasets` for pruning redundant or noisy datasets while curating the suite. ## Step 4: Inner validation and bagging parity The inner validation protocol is one object, `ValidationProtocol` (`tabarena.benchmark.validation_protocol`), owned by the arena context. `TabArenaContext` declares `TABARENA_V0PT1_VALIDATION_PROTOCOL` (8 bagging folds x 1 set, plain stratified splits) and `BeyondArenaContext` declares `BEYONDARENA_VALIDATION_PROTOCOL` (8x1, 5 folds x 5 sets at or below 500 training group instances, task-specific inner splits, class-adaptive folds). `build_jobs` stamps the protocol onto every bagged experiment and refuses one that carries another protocol unless the context was built with `official_validation_protocol=False`. Declare your arena's protocol the same way, on your context subclass (Step 10): ```python from tabarena.benchmark.validation_protocol import ValidationProtocol class MyArenaContext(AbstractArenaContext): benchmark_name = "MyArena" OFFICIAL_VALIDATION_PROTOCOL = ValidationProtocol( tiny_num_bag_folds=5, tiny_num_bag_sets=5, tiny_max_group_instances=500, task_specific_validation=True, adapt_num_folds_to_n_classes=True, name="MyArena", ) OFFICIAL_BUNDLE_HINT = "MyArenaExperimentBundle" ``` Without a subclass, a bare `AbstractArenaContext(validation_protocol=...)` enforces exactly the protocol it is given. Pick a protocol and keep it: an implicit, size-dependent bagging behavior is not a reproducible protocol and makes historical and new numbers incomparable in ways that are hard to detect later. Every result records the protocol it ran under (`results["validation_protocol"]`), the processed `MethodMetadata.validation_protocol` carries its key, and `context.validation_protocol_status` tells official from custom results, so a change of protocol is visible rather than silent. If an earlier version of the domain benchmark ran without explicit bagging control, say so in its changelog rather than presenting old and new numbers as comparable. The task's `group_on` / `time_on` reach the inner folds only under a protocol with `task_specific_validation=True` (BeyondArena's; TabArena-v0.1's has it off). A grouped domain benchmark run under the TabArena protocol therefore leaks groups across bagging folds while its outer splits look correct. Declare a task-specific protocol for grouped or temporal data. With task-specific validation, `resolve_validation_splits` in `tabarena/benchmark/exec_models/autogluon_utils.py` builds group-disjoint or forward-in-time inner folds and passes them to AutoGluon as custom splits. The grouped branch imports Data Foundry, so install `tabarena[data-foundry]` (part of `[benchmark]`). Fold counts adapt at run time and the adaptation is recorded in the result (`clamps`): fewer groups than folds clamps the fold count and sets one repeat, a minority class smaller than the fold count does the same, and a `time_on` task always uses one repeat. Two further execution modes exist for models that carry their own validation: `holdout_experiments=True` keeps a single task-aware holdout split without bagging, and `outer_experiments=True` fits once on all training data with no validation split (the mode most foundation models are benchmarked in; see `examples/beyondarena/advanced/run_quickstart_beyondarena_without_bagging.py`). Neither is an official flavour: they run outside the protocol by construction, need no flag, and their results are recorded as such (`holdout:<key>` / `outer`). A method run this way still gets the shared outer protocol and scoring, but no HPO simulation. ## Step 5: Pick the models and define your bundle Enumerate the zoo from the registry instead of hardcoding names: ```python from tabarena.models import get_model_registry for key, info in get_model_registry().items(): print(key, info.method_metadata.display_name, info.method_metadata.compute, info.pip_extra) ``` `tabarena.models.utils.get_configs_generator_from_name("TabM")` returns a model's generator, whose `model_cls` and `manual_configs[0]` are what `examples/advanced/run_tabarena_model_on_your_data.py` uses to fit a TabArena model outside the benchmark. `tabarena.models.prefetch.prefetch_weights([...])` downloads foundation-model checkpoints before a cluster run. Three bundles ship; pick by protocol, not by name: | Bundle | Random configs | Preprocessing | Task-aware validation | Time limit | |---|---|---|---|---| | `TabArenaV0pt1ExperimentBundle` | 200 | AutoGluon `"default"` | off | 1 h | | `TabArenaExperimentBundle` | required field | `preprocessing_pipelines` required | on | 1 h | | `BeyondArenaExperimentBundle` | 25 | `"tabarena_default"` | on | 4 h | A domain benchmark should subclass `TabArenaExperimentBundle` and pin its own defaults, so every run script shares one protocol: ```python from dataclasses import dataclass, field from typing import ClassVar from tabarena.benchmark.experiment import ModelConstraints, TabArenaExperimentBundle @dataclass(kw_only=True) class RamanBenchExperimentBundle(TabArenaExperimentBundle): n_random_configs: int = 50 preprocessing_pipelines: list[str] = field(default_factory=lambda: ["default"])