| name | compare-runs |
| description | Compare experiment results across servers — differing hyperparameters and final metrics from local wandb data, with dashboard links. Use for "which ablation won" or comparing runs of a project. |
Compare runs
Input: a project name/dir (or infer from context). All read-only.
-
Find the runs: list_sessions cwds point at project dirs;
list_wandb_runs on each host+dir that has the project lists recent runs
with dashboard URLs.
-
Pull each run's local records (newest ~10 unless told otherwise) via
run_command:
- final metrics:
cat '<run>/files/wandb-summary.json'
- hyperparams:
sed -n '1,80p' '<run>/files/config.yaml'
- if the summary is missing, classify crashed/finished from the tail of the
run's logs instead.
Note: wandb-summary.json exists even for offline/unsynced runs — that's
why we read files instead of the wandb API; no key needed.
-
Build the comparison: run (wandb id or dir date) · host · only the
hyperparams that DIFFER between runs (drop identical ones) · the key final
metrics the summaries share (loss/acc/mAP/…) · state · wandb URL when
found.
Report
The table, then: the best run by the metric that matters (ask if ambiguous),
and flag runs that crashed too early to be comparable. If the same config ran
on different hosts with different results, call that out — it usually means
code or data skew (suggest /sync-code).
