understudy-agent-tools

Use when a developer wants to compare candidate models — any mix of local, gateway, or frontier — on the same eval and see quality, latency, cost, and reliability side by side. "Which model should I use", "sweep these models on my benchmark", "compare Gemma vs the frontier on my eval". To stand up and serve a local candidate first, use run-local-model-lab.

Use when a developer's agent — a multi-turn tool-calling loop — should get cheaper, faster, or better. "My agent is too slow", "this workflow costs too much", "test a cheaper model in my tool-calling loop", "A/B the policy model". Covers read-only search loops and state-mutating API workflows alike.

Use when a coding agent must operate the full benchmark lifecycle over local benchmark dirs — "build a benchmark from my traces and run models on it", "review and calibrate the eval", "queue a prompt experiment", "is an executor running", "read the rigor report". Covers traces → build-benchmark → review/feedback → calibration floors → candidate and prompt-override runs → rigor/CI reading → app-replay regression, via the benchmarks MCP server or CLI verbs, plus the run-executor daemon lifecycle.

Use to build a simulated, seeded environment (AutomationBench / verifiers style) so any model can run a captured agentic workload end-to-end and be scored on final state — "simulate this workload's tools", "build a validator for these traces", "let a small model attempt the whole task", "score recall/precision against gold", or any handoff from understand-workload toward whole-case model comparison.

Use to acquire, cache, organize, and explain local open-weight models — "download a model", "what models do I have", "where did the weights go", "free up model disk", "which Gemma/Nemotron should I pull", "how do open models work". Covers where weights come from and live, formats/quantization, gated weights and HF tokens, disk budgeting, start-small-and-cache, and the local→cloud graduation path. American families (Gemma 4, Nemotron 3). To score a local model on a workload, use run-local-model-lab.

Use when a user asks what Understudy is, how Understudy Desktop works, how local model serving, Fusion sidekick, evals, model candidate results, rollout labs, Product Knowledge, or Understudy product capabilities should be explained to developers, customers, or agents.

Use when a developer wants to stand up and run a local model on Apple Silicon against their real workload — "run this model on my Mac", "is a local model good enough before I pay for hosted". Covers the MLX serving rig, scored real-workload evals, and the route decision. For comparing many candidate models on one eval, use compare-model-sweep.

Use when a developer asks a coding agent to improve an LLM app or agent — "make my LLM app cheaper/faster", "raise quality or reliability", "compare models", "pick a model or route". Orchestrates trace → evaluate → optimize (GEPA, automatic prompt evolution) → compare → deploy via worker skills. Not for generic coding unless LLM behavior, cost, traces, evals, or routing is involved.

Use when a coding agent has edited a user's LLM app and wants a regression verdict on the frozen benchmark tasks — "did my code change regress the eval", "run my actual app against the benchmark", "author an app-harness.json". Drafts the understudy.app_harness.v1 sidecar from the user's repo, queues an app_replay run, and reads the honest (marked-anomaly, never fabricated) rows.

Use when a developer's LLM app is running but has NO traces yet and wants capture flowing in about a minute with no code changes — "instrument my app", "start capturing my LLM calls", "I have no traces, get me some", "turn on tracing for my agent". Detects the provider SDK, redirects it through the Understudy gateway (capture on by default) via env vars, verifies a capture actually landed, then hands off to ingest-traces.

Use when a developer wants to build an eval from their real LLM app before changing anything — "measure how my app is doing today", "build an eval from my workload", "we have no baseline", "is my current model actually good". Turns the workload into auditable local artifacts (harness, metric, frozen splits, baseline); has a public-benchmark on-ramp when no traces exist.

Use when a developer has a measured eval and wants the prompt or route improved without retraining — "optimize my prompt against my eval", "close the quality gap with the cheaper model", "squeeze cost without losing quality". Keeps holdout sealed, runs GEPA (automatic prompt evolution) on train/dev only, and requires a claim packet before any savings statement.

Use after a route decision exists and a candidate model must take live traffic safely — "ramp this route", "move 25% of traffic to the new model", "did the route change regress anything", "roll this back", "prove the savings are real". Pre-ramp stability gates, a staged traffic ladder through the Understudy gateway dial, routed-vs-passthrough verification from captures at each step, and explicit rollback triggers.

Use when a developer already has production LLM traces — a bucket of captures, provider log exports, or gateway capture files — and wants them visualized, turned into local redacted eval sets, or profiled for cost. "Visualize this trace", "ingest my traces", "turn these logs into an eval set", "where is my LLM spend going".

Use when a developer wants a captured prompt, trace, or dataset explained before changing anything — "explain this prompt", "what is this trace actually doing", "decompose this workload", "help me understand this dataset before testing models". Builds a shared mental model (purpose, inputs, tools, success criteria) with the user through Q&A.

Use when a workload must learn multi-step behavior through hosted reinforcement-learning (RL) training that local optimization cannot deliver, and needs to become partner-ready. "My agent needs RL", "can we train this policy", "package my environment for a training partner", "is this workload ready for RL". Decides first, then authors the trainable environment, then packages it — never trains here.

Use when a developer asks "is Understudy causing my errors", "which workloads are routed", "is my routing config actually taking effect", "is my provider healthy", "are there 500s on staging", "what's our error rate", "where is my gateway spend going", or wants self-service diagnostics without asking the team. Reads the hosted reporting endpoints with the developer's sk_* key.

Use when a developer wants to run inference or evals through the Understudy gateway — "route my app through Understudy", "set up my account and keys", "A/B a model on part of my traffic" — or must choose between local provider keys and the gateway route ("should I use my OpenAI key or the gateway"). Handles login, projects/keys, model routing, and runs.

Use when you need to know HOW two model runs differ behaviorally on the same tasks, not just THAT one scores higher — per-task trajectory diffing that classifies the gap as persistence/recovery, knowledge, or format/parsing. "why does the bigger model pass these", "is this gap RL-shaped", "diff these two trajectory runs", "where do the trajectories diverge", "what would distillation buy me". The behavioral complement to compare-model-sweep.

Use when a developer wants to replace an expensive frontier model on a classification workload (binary, multi-class, multi-label, or structured extraction) with a fine-tuned open-weight student — "distill this classifier", "can a small model do this tagging job", "the frontier labels these for $X, make it cheaper", "consensus-label my data". Multi-teacher majority-vote labeling, failure-directed SFT data, and a four-way promote/shadow/collect/stop verdict.

Use when a developer wants to fine-tune or distill a local open model on their Mac and see which training method actually moves a captured workload — "fine-tune a local model", "distill the frontier into a small model", "would training close the gap". Compares baseline, rejection-sampled fine-tuning, distillation, and surprisal-gated pedagogical arms; includes learning from privileged answers/feedback.

Use when a developer wants to cut Claude or Anthropic API spend: "lower my Anthropic bill", "audit my Claude spend", "find prompt cache failures", "why is cache_read zero", "can we move this from Claude to OpenAI or a local model". Audits call sites, tokenizer risk, cache structure, batchability, and route candidates before any code edits.

Use as the engaging first-run experience right after the Understudy plugin is installed, or whenever a developer says "get started", "set me up", "I'm new to this", "onboard me", or asks what Understudy is and where to begin. Backgrounds a small open-model download while it profiles the machine, detects ML tooling, interviews the user to gauge experience and goals, and writes a durable ~/.understudy/profile.json so every later skill can meet the user where they are. Hands off to the understudy orchestrator.

Use when a developer wants to run a hosted job — a fine-tune, an RL (reinforcement-learning) run, batch inference, or large-scale trajectory generation — and asks where, how long, and how much. "What would it cost to fine-tune this", "which provider should I use for this RL run", "is this cheaper locally or in the cloud". Estimates and routes; never spends.

Use when a model-level change — a model swap, route change, prompt or playbook edit — is about to ship and the developer wants confidence before prod: "will this work in prod", "run a simulator before I flip traffic", "gate model changes like a pre-commit check". Replays frozen tasks offline, scores output contracts, emits a launch verdict.

Use to give a developer an immediate local-vs-frontier model comparison before they have their own traces — "what can a local model do", "is a small model good enough", "compare a local model to a frontier model", "the onboarding climb". For comparing many models on a user's own eval, use compare-model-sweep; to serve a local model, use run-local-model-lab.

Use when a developer wants a small or local model to take over an agentic task a frontier model one-shots — "can a small model do this whole case", "the prompt is too big for the small model", "hill-climb the local model on this task". Decomposes the task into bounded, flat-context steps behind the same call contract; includes the training path.

Use when a developer wants to install, enable, update, reinstall, remove, or verify Understudy in a coding agent - "install Understudy", "add the Understudy skills", "make Claude/Cursor/Codex/OpenCode/Hermes/Devin see the skills". Chooses the requested agent platform and routes to the local install, reload, onboarding, and uninstall steps.

Use when a developer needs to compress a local model for tool-calling workloads — "which quantization should I use", "why does my model emit broken JSON", "how do I get better tool-call fidelity from a 4-bit model", "OptiQ vs QAT vs naive", "the model stopped calling tools after quantization". Covers layer-aware sensitivity-driven compression, QAT group-size matching, and outcome-optimized calibration for structured output.

Use when a developer wants an always-on ops watcher over logs, command output, or endpoints — "every five minutes look at my logs and tell me what's wrong", "watch this log file and flag anomalies", "review my events on a schedule without burning tokens". Deterministic hash-gated triggers fire a cheap model review only on change, and every review is captured as an eval row for a future fine-tune.

Use when a developer wants to inspect Anthropic or Claude billing evidence from email, invoices, usage exports, or an authenticated billing website: "scan my Anthropic bills", "check billing emails", "look at the console usage page", "find spend hotspots from receipts". Optional, approval-gated evidence source for lower-Anthropic-bill audits.

Use when a developer wants to share how much Understudy saved them, submit an anonymous "lower my Anthropic bill" result, prepare a leaderboard receipt, or send savings metrics back to Understudy. Builds a metrics-only payload from value-report or claim artifacts, redacts identity by construction, and posts only after explicit approval.

Compatibility shim for OpenCode installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify Understudy in OpenCode - "install Understudy in OpenCode", "add the OpenCode skills", "make OpenCode see the skills". Route to install-agent-adapter with platform opencode.

Compatibility shim for Codex installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify the Understudy Codex plugin - "install Understudy in Codex", "add the Codex plugin", "make Codex see the skills". Route to install-agent-adapter with platform codex.

Compatibility shim for Cursor installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify the Understudy Cursor plugin - "install Understudy in Cursor", "add the Cursor plugin", "make Cursor see the skills". Route to install-agent-adapter with platform cursor.

Compatibility shim for Claude Code installs. Use when a developer asks to install, update, enable, reinstall, remove, or verify the Understudy Claude Code plugin - "install Understudy in Claude", "add the Understudy skills", "why can't Claude see the skill". Route to install-agent-adapter with platform claude-code.

Use to turn loose per-task trajectory JSONs (a Lilac export or local rollout corpus) into a queryable, provenance-tracked, contamination-safe dataset that hard-blocks any selection leaking frozen dev/holdout into a train/RL/distill pool. "curate my trajectories", "which rollouts are train-safe", "exclude the holdout rows before RL", "is this distill pool contaminated", "make a hash-stamped selection", "stop hand-filtering trajectories in bash".