| name | benchopt-parallel |
| description | How to scale a benchopt run from local cores to a cluster: choosing a backend (loky/dask/submitit), the --n-jobs vs --parallel-config split, the YAML config for SLURM and Dask, per-solver/per-run SLURM overrides, how --timeout maps to slurm_time, and how caching behaves across nodes and machines. Use when a run is too slow, needs a cluster, or behaves oddly under parallelism — not for the basics of selecting solvers/datasets (see benchopt-run-benchmark).
|
Running benchopt in parallel
Benchopt parallelises at the granularity of one (dataset, objective, solver)
with unique parameters — each such configuration is an independent task. There
are three backends: loky (local, default), dask, and submitit (SLURM).
Local: --n-jobs
benchopt run . -j 4
- Default is sequential (
n_jobs=1). -j/--n-jobs N runs N tasks at once.
- joblib caps C-level (BLAS) threads to avoid oversubscription, so an individual
parallel run can be slower than the same run sequentially. Do not
compare wall-times measured under different
-j values against each other —
use a sequential run for timing-sensitive comparisons.
Cluster: --parallel-config <file.yml>
A YAML file selects the backend and configures it. The only required key is
backend; everything else is backend-specific. --n-jobs still applies (it is
injected as n_jobs into the config).
benchopt run . --parallel-config config_parallel.yml
SLURM (submitit)
backend: submitit
slurm_time: 01:00:00
slurm_gres: gpu:1
slurm_additional_parameters:
cpus-per-task: 10
account: ACC@NAME
- Install with
pip install benchopt[submitit].
- Each unique configuration is submitted as one job in a job array. Logs land
in
./benchopt_run/ inside the benchmark dir.
- Every
slurm_* key is forwarded to submitit.AutoExecutor.update_parameters,
so any submitit/SLURM parameter is available.
- By default there is no cap on simultaneous jobs; set
slurm_array_parallelism: N to limit concurrency when the scheduler requires it.
Dask
backend: dask
dask_address: 127.0.0.1:8786
- Install with
pip install benchopt[dask]. dask_* keys configure the
Client; with no dask_address, a local cluster is started with --n-jobs
workers. coiled_* keys spin up a Coiled cloud cluster instead.
Timeout interaction
--timeout SECONDS (or 10m/2h) is the per-solver wall-clock budget;
--no-timeout removes it (the two flags are mutually exclusive).- Under
submitit, if you do not set slurm_time, benchopt derives the job's
SLURM time limit as 1.5 × --timeout. If a solver legitimately runs longer
than that (slow startup, large data), set slurm_time explicitly or the
scheduler will kill the job before the solver finishes.
Per-solver and per-run SLURM overrides
The config used for a run is merged in order — benchmark config < solver
static < run-specific — so later layers win:
class Solver(BaseSolver):
name = "gpu-solver"
slurm_params = {"slurm_partition": "gpu", "slurm_gres": "gpu:1"}
parameters = {"slurm_nodes": [1, 2]}
Runs with different SLURM parameters are dispatched as separate job arrays.
Caching across nodes and machines
- Results are cached with
joblib.Memory keyed on code + parameters, identically
to a sequential run. On a cluster this "just works" only with a shared
filesystem across the nodes — otherwise each node caches locally.
--collect reads already-cached results without launching new work (useful to
build the report while a big array is still running); it forces the local
loky backend regardless of the config.- To combine results produced on machines without a shared filesystem, run
each separately and
benchopt merge the parquet outputs (see
benchopt-results).
Doc links
