cruijff_kit

Plan LLM fine-tuning and evaluation experiments. Use when the user wants to design a new experiment, plan training runs, or create an experiment_summary.yaml file.

Create custom inspect-ai evaluation tasks through interacted, guided workflow.

Open-ended, Claude-driven exploration of a completed experiment — you decide per-experiment which figures matter and what the results mean. Optional and runnable any time after run-experiment (summarize-experiment is the required post-run step). Produces "Claude's Exploration": a non-deterministic report of what you examined and concluded, with an audit log.

Execute the complete experimental workflow - model optimization followed by evaluation - for all runs in a scaffolded experiment. Use after scaffold-experiment to submit jobs to SLURM.

Archive a completed experiment, preserving all experiment files while deleting bulk checkpoint artifacts. Use after summarize-experiment or explore-experiment when an experiment is complete and results have been reviewed.

Set up complete experimental infrastructure for all runs in a designed experiment. Orchestrates parallel generation of fine-tuning configs (via scaffold-torchtune) and evaluation configs (via scaffold-inspect). Use after design-experiment to prepare configs before running experiments.

Create the standard summary of experiment results from a completed (fine-tuned and evaluated) experiment. Run this right after run-experiment as the required post-run step — captures key metrics (loss, accuracy, regression metrics) into summary.md.

Convert a markdown file to PDF using pandoc. Works with any markdown; commonly used for experiment artifacts (explorations, summaries, reports), e.g. after explore-experiment to create a shareable PDF.

First-time setup for new cruijff_kit users — walks through claude.local.md configuration. Re-run for a health check on an existing config.

Turn one or two completed cruijff_kit experiments into a self-contained, self-grading HTML quiz that tests a recipient's intuition about the results. Use whenever the user wants to teach, share, or stress-test understanding of an experiment — phrases like "make a quiz from this experiment", "test my intuition on these results", "build a quiz to share with the team". Each question embeds everything needed to answer it; submitting reveals the correct answer with an explanation citing the exact row, figure, or section in the experiment artifacts.

Convert tabular data (CSV, Stata, Parquet, etc.) to textual representations for LLM fine-tuning and evaluation. Supports dictionary, narrative, and LLM-generated formats with configurable perturbations.

Create a schema YAML file for tabular source data. Defines columns, types, value maps, and perturbation metadata. Use before design-experiment or convert-tabular-to-text when working with tabular data.

Weekly release check — review changes since last tag and optionally cut a new release.

Create weekly software meeting agenda in the wiki repo.