tune-model-evals

name	tune-model-evals
description	Scientific model and eval tuning loop for prompt/skill/system-prompt changes, model configuration, runtime options, and local/Ollama model sizing or environment variables. Use when tuning eval performance, comparing local or remote models, improving tool-use behavior, reducing timeouts/loops, or deciding whether an eval optimization is worth keeping.

Tune Model Evals

Use this skill when changing anything that might affect model or eval behavior: prompts, product skills, system prompts, model refs, temperature/token settings, tool availability, runtime options, Ollama model size, or Ollama environment variables.

The discipline is: baseline, hypothesis, isolate, run, compare, keep or revert.

Tuning Subcommands

Use these subcommands to run a tuning loop when the user asks to improve a specific or custom dimension such as quality, cost, speed, tool-use reliability, refusal rate, evidence grounding, or loop avoidance. Treat explicit constraints from the user as hard constraints. If constraints are not given, use the common constraints below.

Subcommands can be requested as plain language or exact commands:

• eval:tune research or eval:tune analyze: inspect the current eval, target dimension, constraints, baselines, recent runs, and failure modes before proposing changes.
• eval:tune hypothesis: write one falsifiable hypothesis and acceptance criteria.
• eval:tune experiment: apply the smallest change needed to test the active hypothesis and run the narrowest useful eval matrix.
• eval:tune analyze-result or eval:tune analyze: compare baseline and experiment outputs, including raw artifacts and failure modes.
• eval:tune research-failure: investigate a concrete model/provider/harness failure against current docs, changelogs, source, GitHub issues, and comparable agent projects before spending more live-eval budget.
• eval:tune conclude: decide keep, revert, combine later, or repeat with a sharper hypothesis.
• eval:tune repeat: start the next loop from the conclusion, preserving the same target dimension and constraints unless the user changes them.

When command names collide, infer from context: before an experiment, analyze means research/analyze; after a run, analyze means analyze-result.

Common Constraints

Use these defaults unless the user provides stricter or different constraints:

• No benchmark-answer, hidden-gold, vulnerable-route, payload, prior-finding, or target-specific hint leakage into candidate-visible prompts, skills, memory, RAG, tool descriptions, screenshots, or config.
• Keep improvements generic across targets in the same class.
• Preserve approval gating, target scope, and Evidence requirements.
• Do not regress toolCalls/maxToolCalls reporting, artifact capture, diagnostics, or refusal/blocked-state visibility.
• Prefer cheaper and faster runs while tuning, but do not keep a change that only improves cost or speed by hollowing out useful target work.
• Product tests should prove behavior, not exact prompt text or tuned model defaults.
• For every new or repeated model/provider/harness failure, run an external failure-research pass before more expensive reruns unless the next rerun is only to reproduce the signature cheaply.

Dimension Contract

Every loop must name:

• Primary dimension: the thing being improved, for example quality, cost, speed, tool-call validity, evidence grounding, coverage breadth, timeout rate, refusal accuracy, or onboarding drift.
• Guardrail dimensions: metrics that must not regress, such as authorization safety, contamination risk, cost ceiling, runtime ceiling, evidence count, toolCalls/maxToolCalls, diagnostic preservation, or finding precision.
• Constraints: explicit user constraints plus the common constraints above.
• Budget: max runs, max wall time, cost ceiling, model set, target set, or "stop after one loop" when not specified.

eval:tune research / eval:tune analyze

Use this before changing anything.

Inputs:

• user-requested primary dimension and constraints,
• eval scripts, scenario ids, target, model set, current prompt/config/skill state,
• latest baseline and experiment outputs if available,
• dirty worktree and staged changes.

Actions:

1. Identify the exact eval path and command that measures the primary dimension.
2. Check whether a baseline exists for the same commit, target, model, runtime, scorer, timeout, tool budget, and reset mode.
3. Inspect raw responses, artifacts, scorer notes, tool traces, and errors for the top failure mode.
4. Decide whether the next step is rerun baseline, write a hypothesis, or ask for missing constraints.

Output:

Tuning target:
- Primary dimension:
- Guardrail dimensions:
- Constraints:
- Budget:

Current evidence:
- Baseline:
- Recent runs:
- Dominant failure mode:
- Missing data:

Next command:
- eval:tune hypothesis | eval:tune research-failure | rerun baseline | ask for constraints

eval:tune research-failure

Use this whenever a run exposes provider errors, malformed or dropped tool calls, reasoning/usage/cost metadata anomalies, repeated tool loops, timeouts, missing assistant text, strict real-LLM evidence failures, stream projection mismatches, package-version churn, or adapter-specific behavior. The goal is to discover credible fixes, workarounds, upstream confirmations, or cross-project patterns before burning more model budget.

Inputs:

• exact failure signature and error text,
• model URI, actual provider/model identity, package versions, AI SDK/Mastra/adapter versions,
• result path, raw stream path, Mastra/app DB evidence, trace/span ids,
• current code path and any recently changed config or package versions.

Search priority:

1. Current official docs and changelogs for the package or provider involved.
2. Source and issues/discussions in the primary upstream repo, for example AI SDK, OpenRouter provider, Mastra, model-provider SDKs, or the relevant MCP/browser adapter.
3. Similar reports in active agent projects and harnesses, especially LangChain/LangGraph, Pydantic AI, OpenHands, opencode, Roo Code, TanStack AI, Hermes/OpenFang, ScienceAgentBench/SciBench-style harnesses, SWE-bench/OpenHands benchmarks, and lm-evaluation-harness.
4. Community posts only as weak leads. Never treat blogs, Reddit, or one-off snippets as sufficient evidence without repo/source confirmation or a local repro.

Search query templates:

"<exact error>" "<package or provider>"
"<model/provider>" "tool call" "AI SDK" GitHub issue
"<adapter package>" changelog "<symptom>"
site:github.com <project> "<symptom>" "tool" "retry"
site:github.com <project> "<symptom>" "max iterations" OR "loop"
site:github.com <project> "<provider>" "<model>" "tools"

Evidence grading:

• confirmed-upstream: primary repo issue, merged PR, changelog, official doc, or source code clearly matches our signature/version.
• probable-upstream: multiple independent repo issues or comparable projects report the same package/provider/model signature.
• local-harness-bug: our stream, trace, DB, or code path contradicts the provider behavior or drops valid events.
• model-behavior: provider delivered valid events, but the model repeatedly chooses invalid/no-op behavior.
• unverified-lead: interesting but not enough to act on without a repro.

Allowed outcomes:

• small, reversible patch such as version upgrade/downgrade, provider-option change, try/catch around flaky parsing, stricter diagnostics, loop guard, cost fallback, tool-schema simplification, or adapter swap behind a flag;
• stop the eval batch and put the finding at the top of the report when a credible upstream/package issue could invalidate more rows or inflate cost;
• rerun one cheap reproduction after patching or after pinning a suspected version;
• ask the user before adopting a new package, replacing an adapter, disabling a provider feature, or changing a user-visible model path.

Safety rules:

• Do not run random scripts, install unreviewed packages, paste opaque patches, or trust issue comments without reading the code path.
• Verify installed versions locally before applying online advice.
• Prefer official package upgrades/downgrades, documented options, and minimal local guards over large dependency swaps.
• Preserve raw diagnostics and cost data; do not hide provider errors as model-quality failures.
• Do not contaminate candidate-visible prompts with target-specific failure notes or benchmark answers. Failure research belongs in harness analysis, reports, tests, docs, or scorer diagnostics.

Output:

Failure research:
- Signature:
- Local evidence:
- Classification: confirmed-upstream | probable-upstream | local-harness-bug | model-behavior | unverified-lead
- Sources:
- Candidate fixes/workarounds:
- Risk:
- Decision: stop evals | patch then rerun | keep running with warning | ask user
- Report headline:

eval:tune hypothesis

Write exactly one testable hypothesis. Do not bundle unrelated changes.

Output:

Hypothesis:
If we change <one thing>, then <primary dimension metric> should improve by <margin> because <observed failure mode>.

Change surface:
prompt | skill | system prompt | tool schema | model config | runtime | Ollama env/model size | scorer/reporting

Baseline:
<path/id/command and key metrics>

Acceptance:
Keep only if <primary metric> improves by <margin> within <constraints> and guardrail dimensions do not regress beyond <margin>.

Experiment command:
<exact command(s)>

Regression checks:
<adjacent scenario/model/target or "none for one-loop exploration">

eval:tune experiment

Apply only the active hypothesis and run the narrowest useful eval matrix.

Actions:

1. Confirm the baseline is fresh or rerun it first.
2. Make the minimal code, prompt, skill, config, runtime, or env change.
3. Run the exact experiment command.
4. Save the command, environment, result path, raw output path, and exit status.
5. Stop if cost, runtime, target scope, approval policy, or contamination constraints are breached.

Output:

Experiment run:
- Change:
- Command:
- Result path:
- Exit status:
- Runtime:
- Cost/tokens:
- toolCalls/maxToolCalls:
- Constraint breaches:

eval:tune analyze-result / post-run eval:tune analyze

Compare the experiment against the baseline using like-for-like fields.

Required checks:

• score and normalized score delta,
• primary dimension delta,
• guardrail dimension deltas,
• findings/signals and whether they are Evidence-backed,
• artifact count and key paths,
• toolCalls/maxToolCalls,
• invalid tool calls, loops, repeated calls, dead-end circling, and generic onboarding drift,
• elapsed runtime, timeout status, token counts, cost,
• errors, refusals, blocked states, and diagnostics,
• raw response excerpt and whether the answer was synthesized from Evidence.
• external failure-research status for every new or repeated failure signature.

Output:

Metric	Baseline	Experiment	Delta	Keep?

Then list:

• strongest improvement,
• strongest regression,
• contamination or generality risk,
• raw artifact evidence,
• upstream/docs/GitHub findings and whether they justify stopping more evals,
• recommended conclusion.

eval:tune conclude

Make one decision:

• Keep when the primary dimension improved enough and guardrail dimensions stayed within constraints.
• Revert when it failed, leaked target-specific hints, reduced generality, worsened safety, or only improved a weak proxy.
• Combine later when the effect is promising but should be tested with another isolated change.
• Repeat when data is inconclusive, noisy, underpowered, or reveals a sharper failure mode.

Output:

Decision:
keep | revert | combine later | repeat

Reason:
<one paragraph tied to metrics, raw artifacts, cost, runtime, and constraints>

Changes kept/reverted:
<files/config/env>

Next loop:
<next primary dimension or next hypothesis>

eval:tune repeat

Start the next loop without losing scientific control.

Actions:

1. Carry forward the primary dimension and constraints unless the user changes them.
2. Promote the previous conclusion into the new baseline only if the decision was Keep.
3. If the previous result was noisy, repeat baseline and experiment before changing another variable.
4. Choose the next highest-value failure mode from raw outputs, not from aggregate score alone.
5. Return to eval:tune hypothesis.

Guardrails

• Do not leak benchmark answers, prior findings, target-specific routes, payloads, account names, hidden scorer labels, or tuned hints into candidate-visible prompts, skills, memory, RAG, tool descriptions, screenshots, or config.
• Tune generic behavior: instruction clarity, evidence contracts, stop conditions, tool schemas, tool budgets, model settings, runtime limits, and local model environment.
• Separate model-comparison evals from prompt-improvement evals. Semi-directed strategy prompts belong in prompt-improvement/judge evals, not direct model comparisons.
• Every result summary must include toolCalls/maxToolCalls.
• Prefer path-independent examples and category lists over specific strings, routes, packages, products, payloads, or CVE names.
• Do not add low-value product tests that lock exact tuning knobs such as temperature, max tokens, reasoning effort, model id, or query-string ordering. Put tuning values in eval metadata and experiment reports. Product tests should assert outcomes: artifacts saved/vectorized, memory loaded, tools called or gated, streams projected, Render JSON emitted, findings grounded, and diagnostics preserved.
• Make LLM calls mockable at the model/provider boundary. In Mastra chat-model tests, use EXPLOIT_HUNTER_MOCK_LLM=true or EXPLOIT_HUNTER_LIVE_LLM=false with setMastraChatModelFactoryForTests. Use mocks mainly for response parsing, tool-call handling, stream behavior, refusals/errors, and persistence side effects. Live LLM evals must be opt-in and should default to cheap models such as local gemma4, DeepSeek V4 Flash, GPT OSS 120B, or Gemini Flash/Flash Lite-class models.

External Failure Research

Treat live eval failures as research tasks, not just rows in a table. If a model run fails or behaves strangely, first reduce the local signature, then search for current upstream evidence and known workarounds.

High-value signatures include:

• OpenRouter or provider warnings about reasoning details, signatures, encrypted reasoning, missing usage, or malformed stream chunks.
• AI SDK errors such as no-such-tool, invalid tool input, tool-call repair failures, tool-call validation failures, missing tool results, dropped tool-call events, or finish_reason=tool_calls with no parsed tool calls.
• AgentController/Mastra projection issues such as unmatched tool-call ids, missing assistant text, missing model-inference spans, or background-task chunks not projected.
• Provider/model-specific failures: Gemini/Gemma tool loops, Kimi/Qwen tool-call drift, GPT OSS provider validation errors, or DeepSeek reasoning-only outputs.
• Harness failures: target cleanup, max-iteration stops, stale workspaces, repeated identical tool calls, cost fallback gaps, stream files missing, or strict real-LLM evidence failures.

Research sources to check:

• official docs/changelogs for AI SDK, Mastra, OpenRouter provider, and direct model-provider SDKs;
• primary GitHub issues/discussions and merged PRs for the exact package/version;
• comparable agent/harness projects such as LangChain/LangGraph, Pydantic AI, OpenHands, opencode, Roo Code, TanStack AI, Hermes/OpenFang, ScienceAgentBench/SciBench-style harnesses, SWE-bench/OpenHands benchmarks, and lm-evaluation-harness;
• community posts only as pointers to verify elsewhere.

Examples of credible patterns to look for:

• AI SDK documents NoSuchToolError, InvalidToolInputError, ToolCallRepairError, try/catch handling, stream error parts, and experimental tool-call repair. Check whether a local repair hook, clearer tool-error surface, or schema simplification is a smaller fix than changing prompts.
• AI SDK loop control distinguishes step limits from individual tool-call limits. If one step can emit many tool calls, add a real call budget or repeated-call guard instead of assuming maxSteps is enough.
• LangChain's middleware model includes model-call and tool-call limits with run-level and tool-specific caps. Use that as a design reference when our evals show runaway tool loops.
• OpenHands-style evaluation loops repeat actions until completion or max iterations, with simulated user responses for non-executable actions. If our agent asks for help repeatedly, add a benchmark-mode simulated response or stop condition rather than letting the run drift.
• Pydantic AI issues show that increasing retries alone can repeat the same bad behavior; malformed tool-call recovery needs actionable error context and bounded retry accounting.
• Hermes/OpenFang-style loop guards hash tool name plus serialized args to detect repeated patterns before the hard iteration cap.
• OpenRouter provider issues and changelogs can identify version-specific reasoning-signature, structured-output-plus-tools, stream lifecycle, and adapter incompatibilities. Check whether upgrading, downgrading, disabling a provider option, or using a direct provider/OpenAI-compatible adapter is safer for the next experiment.

Never blindly apply a workaround found online. Convert it into a hypothesis, verify it against local code and installed versions, then run the narrowest reproduction.

Top-of-Report Findings

Every eval report or summary with credible external failure research must put a short External Research Findings or Run-Blocking Findings block immediately after the top cost/token lines and before aggregate score tables.

Include one row per important finding:

Severity	Failure signature	Local evidence	External status	Candidate action	Sources

Severity:

• blocker: likely invalidates more live eval rows, risks major cost inflation, loses incurred cost data, drops tool calls, or hides provider/harness failures.
• warning: credible upstream/provider issue or workaround exists, but current rows remain interpretable with caveats.
• note: weak lead or non-blocking improvement idea.

If severity is blocker, stop or pause further expensive eval batches until the finding is patched, explicitly waived by the user, or isolated to a non-selected model/provider path. Do not bury this information in footnotes, raw logs, or final paragraphs.

Baseline First

Before each experiment, establish a fresh baseline for the exact scenario, model, target, runtime config, and commit state you will compare against.

Record:

• git commit and dirty worktree state,
• eval id, scenario ids, target, run mode, prompt mode,
• model URI, model size/tag, provider, temperature, max tokens, reasoning settings, context size,
• tool budget and toolCalls/maxToolCalls,
• runtime limits, timeout, concurrency, reset mode, network profile,
• Ollama/local env when relevant: OLLAMA_CONTEXT, OLLAMA_FLASH_ATTENTION, OLLAMA_KV_CACHE_TYPE, OLLAMA_NUM_PARALLEL, OLLAMA_MAX_LOADED_MODELS, OLLAMA_KEEP_ALIVE, selected model tag,
• score, findings/signals, evidence artifact count, URL/surface coverage,
• cost, token counts, elapsed runtime,
• errors, timeouts, invalid tool calls, loop/cycle/dead-end detections.

If the baseline is stale, missing, from a different model/config, or contaminated by a different prompt, rerun it.

Experiment Design

Write a short experiment log before making changes:

Hypothesis:
If we change <one thing>, then <metric> should improve by <margin> because <reason>.

Change surface:
prompt | skill | system prompt | tool schema | model config | runtime | Ollama env/model size

Baseline:
eval id/path, model, target, scenario, metrics

Acceptance:
keep only if <primary metric> improves by <margin> and no guardrail metric regresses beyond <margin>.

Regression checks:
evals/scenarios/models that must not get worse.

Test one hypothesis at a time. Do not bundle changes unless the experiment is explicitly testing a combined hypothesis; even then, run individual-effect experiments first when feasible.

Tuning Surfaces

Prefer changes in this order unless evidence suggests otherwise:

1. Candidate-visible prompt/skill/system prompt wording

   • Make instructions more generic, concrete, and action-oriented.
   • Add evidence contract, output contract, stop condition, and approval boundary.
   • Remove ambiguity that causes onboarding, refusal, or generic help text.
   • Do not add target-specific hints.

2. Tool and runtime shape

   • Reduce tool clutter for small models.
   • Make the first useful action obvious.
   • Tighten tool descriptions and schemas.
   • Tune max steps, timeout, concurrency, reset behavior, and seeded evidence policy.

3. Model configuration

   • Tune temperature, maxTokens, context size, reasoning options, provider options, and model URI.
   • Keep llm://provider/model refs and use existing model config helpers.

4. Local/Ollama configuration

   • Select model size/tag based on available RAM and target latency.
   • Tune env vars such as context size, flash attention, KV cache type, parallelism, loaded-model count, and keep-alive.
   • Run local models sequentially when memory pressure or contention would confound results.

Small Model Hypothesis Seeds

When local or small models produce generic onboarding, weak findings, invalid tool calls, too few tool calls, or mostly baseline-derived signals, start with hypotheses like these. Treat them as seeds, not a fixed list; inspect the observed outputs and invent sharper hypotheses from the actual failure mode.

• Stronger target-work framing: If the prompt explicitly suppresses onboarding and requires immediate target work, then generic-help drift should decrease and first useful action rate should improve.
• Simpler tool surface: If the run exposes fewer tools or simpler tool schemas, then valid tool-call rate should improve and malformed tool calls or tool-schema repair loops should decrease.
• Required first read/probe step: If the prompt or runner requires one concrete read/probe before final answer, then zero-tool runs should decrease and evidence-backed findings should increase.
• Pre-digested local evidence: If local models receive a compact, generic URL inventory or evidence artifact summary, then target-specific reasoning should improve without leaking hidden answers.
• Tool-like text recovery: If a local wrapper detects tool-like text emitted as plain text and retries with a repair instruction, then valid tool calls should increase and dead-end conversational replies should decrease.

For each seed, run it as its own experiment first. Only test combinations after individual effects are measured.

Scientific Loop

For each experiment:

1. Capture or rerun the baseline.
2. State one falsifiable hypothesis and expected movement.
3. Apply only the intended change.
4. Run the same eval matrix needed to prove improvement and guard against regression.
5. Compare against baseline using the same scorer and output fields.
6. Inspect raw responses and artifacts, not only aggregate scores.
7. Decide:
   • Keep when improvement clears the margin and no important regression appears.
   • Revert when it fails, creates contamination risk, reduces generality, or only improves a weak/proxy metric.
   • Combine later only after individual effects are understood.

Metrics To Collect

Required for every run:

• score and normalized score,
• findings/signals and whether they are evidence-backed,
• evidence artifact count and relevant artifact paths,
• toolCalls/maxToolCalls,
• invalid tool calls, tool-schema repair attempts, tool loops, repeated calls, dead-end circling,
• elapsed runtime and timeout status,
• token counts: input, output, reasoning, cached input when available,
• estimated cost and budget spent,
• errors, blocked states, refusal/onboarding/generic-help drift,
• raw response excerpt and whether the answer was synthesized from evidence.

Loop/dead-end indicators include repeated identical tool calls, repeated malformed tool calls, repeated requests for already-provided target context, generic onboarding after a concrete target prompt, and long runtime with zero tool use.

Comparison Rules

• Compare like with like: same target, scenario, run mode, model, timeout, tool budget, reset mode, and scorer unless the experiment is changing one of those fields.
• Use absolute and relative deltas. A tiny score bump with doubled runtime or worse tool behavior may not be worth keeping.
• Treat a finding as weak if it is only inferred from baseline/seed evidence and the model did not produce target-specific reasoning or valid tool use.
• Run at least one adjacent regression scenario when changing shared prompts, skills, system prompt, tool descriptions, or runtime defaults.
• If results are noisy, repeat baseline and experiment runs before deciding.

Reporting Format

End with a compact experiment table:

Experiment	Change	Baseline	Result	Delta	toolCalls/maxToolCalls	Runtime	Tokens	Cost	Decision

Then summarize:

• what improved,
• what regressed,
• whether local/small models used tools or only baseline evidence,
• loop/dead-end patterns,
• kept changes and reverted changes,
• next highest-value hypothesis.

Do not claim success from aggregate score alone. Tie the decision to raw responses, evidence artifacts, tool behavior, cost, and regressions.