Plan LLM fine-tuning and evaluation experiments. Use when the user wants to design a new experiment, plan training runs, or create an experiment_summary.yaml file.
Create custom inspect-ai evaluation tasks through interacted, guided workflow.
Open-ended, Claude-driven exploration of a completed experiment — you decide per-experiment which figures matter and what the results mean. Optional and runnable any time after run-experiment (summarize-experiment is the required post-run step). Produces "Claude's Exploration": a non-deterministic report of what you examined and concluded, with an audit log.
Execute the complete experimental workflow - model optimization followed by evaluation - for all runs in a scaffolded experiment. Use after scaffold-experiment to submit jobs to SLURM.
Archive a completed experiment, preserving all experiment files while deleting bulk checkpoint artifacts. Use after summarize-experiment or explore-experiment when an experiment is complete and results have been reviewed.
Set up complete experimental infrastructure for all runs in a designed experiment. Orchestrates parallel generation of fine-tuning configs (via scaffold-torchtune) and evaluation configs (via scaffold-inspect). Use after design-experiment to prepare configs before running experiments.
Create the standard summary of experiment results from a completed (fine-tuned and evaluated) experiment. Run this right after run-experiment as the required post-run step — captures key metrics (loss, accuracy, regression metrics) into summary.md.
Convert a markdown file to PDF using pandoc. Works with any markdown; commonly used for experiment artifacts (explorations, summaries, reports), e.g. after explore-experiment to create a shareable PDF.