How to add a dataset to an existing benchopt benchmark: the Dataset class contract (get_data feeding Objective.set_data), parameters, requirements, expensive one-time prepare() with caching, reproducible seeding, a zero-dependency Simulated dataset, and test_parameters. Use when implementing or fixing a dataset, not when authoring a whole benchmark.
How to add a solver to an existing benchopt benchmark: the Solver class contract (set_objective/run/get_result), sampling strategies and stopping criteria for convergence curves, parameters, requirements, skipping incompatible problems, warm-up/JIT handling, and test_parameters. Use when implementing or fixing a solver, not when authoring a whole benchmark.
How to author a new benchopt benchmark (datasets, solvers, objective): the component contract, benchmark-wide config and dependencies, data preparation, testing, and CI. Use when creating or restructuring a benchmark repo, not when working on the benchopt library itself.
How to drive a benchmark's code directly from Python via the Benchmark object, without launching the full `benchopt run` CLI: load datasets and the objective, inspect what `Dataset.get_data()` returns, replay a solver (`set_objective`/`run`/`get_result`) and call `Objective.evaluate_result()` on its output or an arbitrary checkpoint. Covers common pitfalls — NaNs, diverging curves (benchopt's divergence guard), and stale-cache surprises plus how to bust the cache — and points to `benchopt test` for catching design problems early. Use when debugging a benchmark's own code (a failing `get_data`, a diverging solver, NaNs, a suspicious metric, results that don't update after an edit) rather than running or authoring one.
How to scale a benchopt run from local cores to a cluster: choosing a backend (loky/dask/submitit), the --n-jobs vs --parallel-config split, the YAML config for SLURM and Dask, per-solver/per-run SLURM overrides, how --timeout maps to slurm_time, and how caching behaves across nodes and machines. Use when a run is too slow, needs a cluster, or behaves oddly under parallelism — not for the basics of selecting solvers/datasets (see benchopt-run-benchmark).
Explore and manage benchopt result files: read run parquets in Python (benchopt.results.read_results), understand the dataframe schema, use the CLI (plot/merge/publish) and outputs/ layout, save custom views with plot_configs, write custom BasePlot plots, and merge results across machines. Use when analysing, slicing, comparing, plotting, or sharing benchmark results.
How to run a benchopt benchmark and manage results: selecting objectives/datasets/solvers (with parameter grids), repetitions and budgets, parallelism (local and SLURM), conda environments, installing requirements, preparing data, caching, and plotting/publishing. Use when running an existing benchmark, not when authoring one.