// Run one Autolab benchmark experiment safely on Hugging Face Jobs. Use when a planner, reviewer, or experiment worker is preparing, auditing, launching, or reviewing a single train.py hypothesis against the current local promoted master.
Use to select models to run locally with llama.cpp and GGUF on CPU, Mac Metal, CUDA, or ROCm. Covers finding GGUFs, quant selection, running servers, exact GGUF file lookup, conversion, and OpenAI-compatible local serving.
Use Hermes delegate_task cleanly in this repo for planner, reviewer, researcher, reporter, experiment-worker, and memory-keeper roles.
Operate the local Trackio reporter for Autolab HF Jobs. Use when a reporter or planner needs to inspect scores, active jobs, worker anomalies, duplicate launches, or the overall experiment board.
Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing repositories, models, datasets, and Spaces on the Hugging Face Hub. Replaces now deprecated `huggingface-cli` command.
| name | autolab-managed-experiment |
| description | Run one Autolab benchmark experiment safely on Hugging Face Jobs. Use when a planner, reviewer, or experiment worker is preparing, auditing, launching, or reviewing a single train.py hypothesis against the current local promoted master. |
| version | 1.0.0 |
| metadata | {"hermes":{"category":"autolab","requires_toolsets":["terminal"]}} |
Use this for any single Autolab experiment that should result in exactly one managed benchmark run.
. ~/.autolab/credentialsuv run scripts/refresh_master.py --fetch-dagtrain.py for the single intended hypothesis.uv run scripts/hf_job.py preflightuv run scripts/hf_job.py launch --mode experimentuv run scripts/hf_job.py logs <JOB_ID> --follow --output /tmp/autolab-run.loguv run scripts/parse_metric.py /tmp/autolab-run.loguv run scripts/submit_patch.py --comment "..."val_bpb beats current master.train_orig.py as the refreshed local-master base. If preflight reports
multiple known hypothesis categories, stop and inspect the diff before
launching.main and origin/main when judging freshness. In this rig
repo those refs describe control-plane history, not the benchmark master. The
comparable base is whatever refresh_master.py just wrote into
train_orig.py, research/live/master.json, and research/results.tsv.uv run scripts/hf_job.py launch --mode prepare from an
experiment-scoped worktree. prepare is shared bootstrap work, not
per-experiment work.uv run scripts/hf_job.py preflight --json
Use this when you need to inspect the diff preview, active conflicts, or
detected change categories programmatically.uv run scripts/trackio_reporter.py summary --max-jobs 25
Use this to confirm the experiment id or hypothesis is not already active.