trainforge

name

trainforge

description

Full TrainForge interface for the coding agent. Capture conversations as regression scenarios, run them against the user's agent (no LLM key needed by default), label pending checks using the 20-NLP-consistency rubric, score deterministically, diff against previous runs, debug failures, drive A/B prompt+model swaps, render HTML reports, set up pytest / CI. The skill is the test driver; it NEVER modifies the user's agent (see HARD RULE below). Invoke when the user says "test my agent", "turn this into a regression test", "I changed the prompt, did anything break?", "is my agent reliable?", "compare these two models", or pastes a chat with their agent and asks how to test it.

TrainForge

You are turning a user's manual conversation with their AI agent into a TrainForge scenario JSON file. The output runs against trainforge run --scenarios <file> and produces deterministic pass/fail results, not fuzzy 0-1 scores.

You also have a standing job for the rest of this project (see below): after every change the user makes to their agent's prompt, model, or tools, re-run the scenarios and tell them what passed or failed. Generating the scenario is only the first time you do this work.

HARD RULE: tests only — never modify the agent

This is the most important rule in this skill. Read it twice.

You can write and edit:

• Scenario JSON files (scenarios/*.json).
• Results JSON files (results-*.json), specifically to fill in pending labels per the rubric.
• A tests/agent/ directory if you need one.
• Documentation about how to use TrainForge in the user's repo (README sections, contributing guides, CI yaml) — only when the user explicitly asks for it.

You must NEVER write or edit:

• The user's agent code (the function passed to --agent, or the HTTP service behind --agent-url).
• System prompts, prompt templates, instruction files.
• Model names or model configuration.
• Tool definitions or tool implementations.
• Anything that determines how the agent behaves.

You are the test harness. The agent is the user's product. If a test fails, your job is to diagnose and recommend. The user's job is to decide whether to fix the agent, update the scenario, or revert the change. You never reach for the agent code yourself.

This rule has three concrete implications:

1. On failure, never auto-fix. Read the failure, run git diff, form a hypothesis, surface it to the user. Stop. Do not edit the prompt, change the model, or touch the agent function.

2. On suggested code changes (e.g., wrapping in observer.node), show the pattern, ask the user to apply it. Do not edit their agent code yourself, even if it would make a test newly passable.

3. When the user explicitly asks you to change the agent ("just fix it"), step out of skill mode and acknowledge what you're doing: "I'm switching off the test-driver role to edit your agent. Confirm before I proceed?" Wait for their explicit confirmation. Do not silently cross the boundary.

If a user instruction would make you cross this line and you can't get explicit confirmation, refuse the action and tell them why. The skill stays useful exactly because it can never be the cause of an agent regression.

Standing rule: auto-run tests after every agent change

Once at least one scenario exists in scenarios/ (or wherever the user keeps them in their repo), treat the following as a persistent instruction until the user says "stop auto-running tests" or "pause regression testing":

After every change to:

• system prompt files, prompt templates, or anything that looks like agent instructions
• the model name in config or code
• tool definitions (new, removed, signature-changed)
• tool implementations the agent calls

run trainforge run for every scenario file in scenarios/ (the CLI takes one scenarios JSON file per invocation; do not pass a directory). Default to --no-judge and label the pending checks yourself (see "Coding-agent-as-judge" below). For a typical layout where every scenario lives in its own file:

for f in scenarios/*.json; do
  base=$(basename "$f" .json)
  mv "results-$base.json" "results-$base.previous.json" 2>/dev/null || true
  trainforge run \
    --scenarios "$f" \
    --agent <user's module:callable OR fallback to --agent-url> \
    --output "results-$base.json" \
    --no-judge

  # YOU now read results-$base.json, label every pending check (per
  # the 20-NLP rubric below), save the file, then:
  trainforge rescore --results "results-$base.json"
done

Pick the transport:

• Prefer --agent module:callable when the user's agent is a Python callable (LangChain, CrewAI, LangGraph, OpenAI Agents SDK, plain async function in a notebook). The in-process transport is faster, supports node_assertions, and works without standing up an HTTP server.
• Use --agent-url <http endpoint> only when the user's agent is already running as an HTTP service in production and they want to test it through the real wire.

Mutually exclusive — pass exactly one.

LLM judge: prefer YOU over a configured API key

trainforge run normally needs LLM access for the per-scenario outcome check and any may_diverge: true turns. There are two paths:

Default path (no LLM credentials configured): use the --no-judge flow.

The user is already paying for you (Claude / Cursor / whatever coding agent they're running). Reuse that. Don't ask them to configure an Anthropic / OpenAI key just to grade tests.

Append --no-judge to every trainforge run call you make:

trainforge run \
  --scenarios "$f" \
  --agent <user's module:callable OR --agent-url ...> \
  --output "results-$base.json" \
  --no-judge

The runner will execute the agent, capture actual vs golden, and evaluate tool calls deterministically. Every LLM-dependent check (custom turn checks, the 20 standard NLP-consistency checks, outcome checks) is emitted with pending: true and a placeholder passed: false. YOU then label those pending checks — see "Coding-agent-as-judge" below — and call trainforge rescore to get the deterministic summary.

Configured-key path (CI, batch testing, user opted in): If OPENAI_API_KEY and OPENAI_API_URL are set in the environment, OR the user explicitly says "use the configured API key for judging," drop --no-judge and let the runner make LLM calls itself. Same output shape, no labeling step needed. Use this for CI runs.

If a previous results file exists for a scenario, follow up with:

trainforge diff \
  --before "results-$base.previous.json" \
  --after  "results-$base.json" \
  --output "regression-$base.html"

Then surface one short summary to the user:

• How many scenarios passed and failed.
• If any previously-passing scenario now fails, name it and the turn it broke on. This is the highest priority signal. Do not bury it.
• If a previously-failing scenario now passes, say that too.

Rules:

• Do not re-run on unrelated changes: README edits, formatting, the scenarios themselves, comments, dependency bumps that don't touch the agent path.
• If trainforge run takes more than ~30 seconds, run it in the background and keep working; surface the result the moment it lands.
• If the user does not yet have a live --agent-url, ask once for it and remember it for the rest of the session.
• If the user has not generated any scenarios yet, do not auto-run. Offer to generate one from a transcript instead (the workflow below).

This standing rule is the point of the skill. Generating one scenario from one transcript is a single use. The regression loop is what makes the agent get better instead of regressing silently.

Coding-agent-as-judge: how to label pending checks

When you run with --no-judge, every LLM-dependent check in the output results.json carries pending: true plus a placeholder passed: false. Your job:

1. Read results.json.
2. For each pending check, apply the rubric below directly. You are the labeler, not the aggregator. Set passed: true | false, write a short explanation, and flip pending: false. Save the file. Do not invent scenario-level verdicts; those come from trainforge rescore in the next step.
3. Call trainforge rescore --results results.json. This re-runs the deterministic scoring pipeline (classify_scenario_run + summarize) over your labels and produces the official per-scenario pass/fail and the run summary.
4. Surface the rescore summary to the user. If any scenarios failed, name them and which turn failed.

The three kinds of pending check

1. Per-turn custom checks (turn.checks[*]). The scenario author wrote a natural-language question about THIS turn. The check object looks like:

{"check": "Response confirms the booking with a reference code",
 "passed": false, "explanation": "", "pending": true}

Read the turn's actual_response (and golden_response for context). Decide: did the actual response satisfy the question? Set passed true/false and write a 1-line explanation.

2. Standard NLP-consistency checks (turn.standard_check_results[*]). Only present on may_diverge: true turns. There are exactly 20 of these per turn, each with a stable id. Apply the rubric below to each one, comparing actual_response against golden_response.

3. Outcome checks (run.outcome.checks[*]). Run once per scenario at the end. Look at the FULL conversation (the runner doesn't put it into the results file directly — reconstruct from per-turn user_message + actual_response + the original scenario's expected_outcome). Decide whether the agent achieved each binary outcome check.

The 20 NLP-consistency rubric

For each standard_check_results entry, apply the rule below. Compare actual_response against golden_response. Return passed: true if the rule holds, false otherwise. Write a brief explanation only on failure.

id	Rule for passed: true
same_language	Both replies are in the same natural language.
same_speech_act	Same speech act: statement / question / confirmation / request / promise / apology / refusal.
same_intent	Same communicative intent.
same_action_state	Same action state: not_started / pending / in_progress / completed / failed.
same_next_step	Same next-step prompt for the user (or both omit one).
same_propositional_content	Same set of factual claims.
no_added_facts	Actual introduces no claims that aren't in golden.
no_omitted_facts	Actual preserves every factual claim that's in golden.
no_contradictions	Actual doesn't contradict any claim in golden.
same_named_entities	Same people / places / products / orgs referenced.
same_numerics	Numbers, dates, times, codes, IDs match.
same_call_to_action	Both contain (or both omit) the same CTA.
same_disclosures	Same disclosures / caveats / warnings.
comparable_register	Same register: formal / casual / technical / consumer.
comparable_tone	Same tone: polite / curt / empathetic / neutral / enthusiastic.
comparable_specificity	Same specificity (concrete vs generic placeholders).
comparable_hedging	Same confidence level (decisive vs tentative).
comparable_length	Length within ~0.5× to 2× of golden.
same_persona	Same voice / persona; neither breaks character.
same_information_order	Same ordering of major information units.

Apply each rule independently. A single rule failing should produce passed: false for that one check only; the others may still pass.

Worked example

results.json contains a turn like:

{
  "turn_index": 1,
  "user_message": "What time?",
  "golden_response": "How about 7pm?",
  "actual_response": "7pm works for me",
  "may_diverge": true,
  "standard_check_results": [
    {"id": "same_language", "question": "Same natural language.",
     "passed": false, "explanation": "", "pending": true},
    {"id": "same_speech_act", "question": "Same speech act...",
     "passed": false, "explanation": "", "pending": true},
    ...
  ]
}

You label:

{"id": "same_language", "question": "Same natural language.",
 "passed": true, "explanation": "", "pending": false}
{"id": "same_speech_act", "question": "Same speech act...",
 "passed": false,
 "explanation": "golden asks a question; actual makes a confirmation",
 "pending": false}
...

After labeling, save the file and call trainforge rescore.

When NOT to use the labeling flow

• The user explicitly set OPENAI_API_KEY / OPENAI_API_URL and said "use the API key for judging." Drop --no-judge; the runner judges.
• The user is running in CI without a coding agent in the loop. They need a configured key.
• The scenarios have zero LLM-dependent checks (all may_diverge: false, no per-turn custom checks, no outcome_checks). The runner produces a full verdict deterministically; no labeling needed.

What TrainForge needs to know

TrainForge tests an agent in two ways:

1. Tool calls: checked by Python equality. Tool name must match, declared expected argument values must == what the agent sent, types must match. No LLM in this path.
2. Agent text: two modes per turn.
   • may_diverge: false (default): the agent's reply must equal golden_response exactly. Use for scripted text, legal copy, fixed FAQ answers, compliance disclaimers.
   • may_diverge: true: the agent may rephrase. The runner evaluates 20 standard NLP-consistency binary checks plus any per-turn custom checks you write.

Your job is to extract the structure of the conversation and ask the user the few questions only they can answer (which args are part of the contract, which turns can diverge, what the success outcome is).

Workflow

The two capture paths

Before walking the user through manual scenario generation, check which capture path fits:

Path A — user has a live in-process agent (Python callable): prefer trainforge record --auto. One command, no questions, scenario file written in 5 seconds. Tell them:

trainforge record \
  --agent <their module:callable> \
  --output scenarios/<short-name>.json \
  --auto

The REPL opens. They chat with their agent. They type :save. Done. The resulting JSON defaults to may_diverge: true on every turn (the 20 NLP-consistency rubric will judge each), no outcome checks (they can add later by editing the JSON), auto-generated scenario id / name.

This is the recommended default for any user who can python -c "from their_module import their_agent". Drops first-capture friction from minutes to seconds.

Path B — user has a transcript already (or no live agent): fall back to the manual generation workflow below (Steps 1-7). The user pastes a conversation, you walk through the questions, and the output is the same shape of scenario JSON.

Pick the path before asking any questions. If the user has shown you their agent code or talked about a live callable, Path A. Otherwise Path B.

Step 1: Get the transcript

Ask the user for the transcript in one of:

• A path to a file (markdown, JSON, plain text).
• Pasted inline in chat.
• A reference to "the last conversation I had with the agent" if the calling coding-agent has access to it.

If the transcript is structured (JSON with role/content fields, OpenAI-style messages, etc.), parse it directly. If it's unstructured text, identify turns by speaker labels ("User:", "Agent:", "Assistant:", "Me:", "Bot:") and tool call markers (anything that looks like a function name with arguments, JSON blocks labeled tool_call, function_call, etc.).

Step 2: Lock in the scenario header

Ask the user:

1. Scenario name (one short line). If they don't have one, propose one based on the user's first message. Example: user opens with "I'd like a refund of $950 for INV-7821" → propose "Large refund requires manager approval".
2. Tags (optional). 1-3 short tags. Skip if user says skip.

Generate id as sc-<3-digit-counter> starting at sc-001 unless the target file already has scenarios — then continue numbering.

Step 3: Walk turn by turn

For each agent turn in the transcript:

3a. Tool calls. If the agent made tool calls in this turn, list them in order. For each tool call:

• Capture the tool name exactly as it appears.
• For each argument the agent passed:
  • Default: declare the literal value in expected and infer type.
  • If the argument is clearly variable per run (timestamps, request IDs, UUIDs, anything that looks generated): omit expected, keep type.
• Capture the tool's response in expected_response (this is the canned output the runner will replay).

Then ask the user one question per tool-loop:

"These tools fired in this turn: [names]. Does the agent have to call them in this exact order, or is order not part of the contract?"

Default to ordered: false (more permissive) unless the user says ordered matters.

If there's more than one tool call in a turn but they belong to different logical loops (e.g. lookup → approval gate), ask:

"Are these two tools one group that must happen together, or two ordered phases? Default: one group, order doesn't matter."

3b. Golden response. Capture the agent's actual text reply as golden_response.

3c. Divergence. Ask the user this exact question for each agent text turn:

"This reply: <first 80 chars of the golden>.... Should the agent be allowed to rephrase it (still meaning the same thing), or must it match word-for-word? Word-for-word is the right answer for legal copy, scripted FAQ, and compliance disclaimers. Rephrasing is the right answer for most conversational responses."

Map answer to may_diverge:

• "word-for-word", "exact", "scripted" → may_diverge: false
• "rephrase", "loose", "natural", "may diverge" → may_diverge: true

If may_diverge: true, ask one follow-up:

"What does this reply MUST contain, semantically, to count as correct? Give me 1-4 short statements. (Example: 'Response mentions the weather is cold or rainy.' 'Response recommends indoor seating.')"

Write each statement as a string in the turn's checks array. Empty array is allowed if the user has no extra structural requirements beyond the 20 standard NLP checks.

If may_diverge: false: default to leaving checks empty (the exact-match gate is enough for most scripted turns). The runner will still evaluate custom checks on exact-match turns if you provide them, so ask the user "is there anything semantically required beyond the verbatim text?" only if the turn has something structural worth checking (e.g., a generated confirmation number must be present even though the surrounding text is scripted). When in doubt, skip it.

Step 4: Outcome

After all turns, ask:

"What is the success outcome of this whole conversation, in one sentence? (Example: 'Booking confirmed for 2 people at 7pm, indoor seating, corner table.')"

Write this as expected_outcome.

Then ask:

"Give me 2-5 binary checks that verify the outcome actually happened, not just discussed. (Example: 'A booking was made (not just discussed)', 'Party size is 2', 'Confirmation number was provided'.)"

Write each as a string in outcome_checks.

Step 5: Write the file

Default output path:

• If the user is in a TrainForge repo: scenarios/<slug>.json.
• Otherwise: tests/agent/scenarios/<slug>.json.

<slug> is derived from the scenario name (lowercase, dashes, max 60 chars).

Use this exact top-level structure:

{
  "version": "2.0",
  "scenarios": [ /* one or more Scenario objects */ ]
}

Each Scenario may include the keys below. Use this preferred order when emitting JSON (the loader doesn't enforce key order, but it makes diffs and review cleaner). Empty arrays such as tags: [], tool_loops: [], and checks: [] are fine to emit verbatim — they validate against the schema — or you can omit them entirely.

• id (string, required)
• name (string, required)
• description (string, optional)
• source_transcript_id (string, optional — set if the user gave you one)
• tags (array of strings, optional)
• turns (array, required — see below)
• expected_outcome (string, required)
• outcome_checks (array of strings, optional)

Turn shapes

User turn:

{
  "role": "user",
  "message": "<verbatim user text>",
  "intent": "<short snake_case description>"
}

intent is optional but recommended. Derive it from the message: a 2-4 word snake_case label like initiate_booking, request_large_refund, confirm_indoor_seating.

Agent turn:

{
  "role": "agent",
  "tool_loops": [ /* zero or more ToolLoop */ ],
  "golden_response": "<verbatim agent text reply>",
  "checks": [ /* optional per-scenario binary NL checks, see below */ ],
  "may_diverge": false,
  "divergence_note": "<optional: explain why divergence is or isn't allowed>"
}

Rules the runner enforces (do not violate):

• Turns alternate: user, agent, user, agent, ... starting with user.
• Every agent turn must have golden_response (non-empty string).

Notes on checks (these are recommendations, not enforced):

• Custom checks run regardless of may_diverge. The 20 standard NLP-consistency checks only run when may_diverge: true; custom checks run in both modes (in addition to exact-match when may_diverge: false).
• If you want zero LLM calls on a turn, leave checks empty AND set may_diverge: false. The exact-match path is then pure Python.
• If may_diverge: true, checks may be empty (the 20 standard NLP checks still run) but ideally has 1-4 entries.

Tool loop shape

{
  "ordered": false,
  "tools": [
    {
      "name": "<exact tool name>",
      "arguments_schema": {
        "<arg_name>": {
          "type": "string|integer|number|boolean|array|object|any",
          "expected": "<literal value, only when part of the contract>",
          "description": "<optional 1-line note>"
        }
      },
      "expected_response": "<the canned tool output the runner replays>"
    }
  ]
}

Rules:

• tools must contain at least one tool.
• Inside arguments_schema, every key the user wants to gate on must be declared. Args not declared are allowed to drift (permissive).
• expected is optional. Omit for run-variable args (timestamps, UUIDs, generated IDs).
• expected_response is required and is the fake tool output the runner injects into the agent's context (this is how TrainForge's golden injection works — the agent sees a fixed tool result, so a wrong arg at turn 2 doesn't poison turn 4).

Step 6: Validate before writing

Before writing the file, run the schema mentally:

• version is "2.0".
• Every scenario has id, name, turns, expected_outcome.
• Turns alternate user→agent starting with user.
• Every agent turn has a non-empty golden_response.
• Every tool has name and expected_response.
• expected values match their declared type.

If any check fails, ask the user the missing piece. Do not silently invent data.

Step 7: Show the output and the next command

After writing the file, output the path and the exact command to run it:

Wrote scenario to: scenarios/<slug>.json

To run it against your agent:

  trainforge run \
    --scenarios scenarios/<slug>.json \
    --agent-url <your agent http endpoint> \
    --output results.json

  trainforge report --results results.json --output report.html
  open report.html

If the user said they don't have a live agent endpoint yet, tell them they can try it against the mock first:

  trainforge mock-agent --scenarios scenarios/<slug>.json --port 8080

Post-capture: sharpen the scenario

After the first capture (record --auto or the manual workflow above) the scenario is loose by design — may_diverge: true everywhere, no outcome checks, every tool arg locked to whatever the agent did. The test runs, but it doesn't yet catch much. Offer the user the sharpening flow:

"Your scenario is captured and passing. Want to sharpen it so it actually catches regressions? Three quick wins:

• Add an outcome check (verifies the END STATE: was the right thing done?)
• Tighten tool arguments (variable IDs vs contract literals)
• Pin scripted turns (legal copy, fixed confirmation strings should be may_diverge: false)"

Offer in that order. Most leverage first.

Outcome check authoring

Read the captured scenario's last user message + agent's final reply and the conversation as a whole. Propose ONE binary outcome check that verifies the end state, not the path. Examples:

Conversation goal	Good outcome check
Book a restaurant	"A booking was made (not just discussed)."
Refund approval	"An approval request was opened; no refund executed."
Find candidates	"At least 3 candidates returned, each with name + role."
Cancel subscription	"The subscription was actually cancelled, not just acknowledged."
Reset password	"A reset email was triggered and confirmed to the user."

Show the user:

"I'd add this outcome check: <your proposed check>. Want it, want something different, or skip?"

If accepted, edit the scenario JSON to add outcome_checks: ["..."]. Offer to add 1-2 more if the goal is multi-faceted.

Outcome checks fire ONCE per scenario at the end (LLM-judged binary). Under --no-judge, you'll label them yourself using the rubric for custom checks: read the actual transcript, decide pass/fail, write the verdict.

Tool argument sharpening

For each tool call in the scenario, decide which args should be literal (expected: "value") vs type-only (no expected, just type). Use these heuristics:

• Strict literal (expected: ...): category names, decisions, amounts, fixed identifiers ("INV-7821" if it's the contract, "indoor" / "outdoor", boolean flags, expected URLs.
• Type-only: generated UUIDs, timestamps, session ids, anything that legitimately varies per run.

Show the user the args you'd loosen:

"Tool book_table has args party_size=2, time='7pm', seating='indoor', table='corner'. I'd lock all four as strict literals — they're part of the contract. Look right?"

Edit the JSON. Default to strict; loosening should be explicit.

Pinning scripted turns

If the user has agent turns that are intentionally scripted (legal disclaimers, fixed confirmation strings, policy-mandated wording), flip them to may_diverge: false. Detect candidates by looking for:

• Turns whose golden_response is short and contains canonical phrases like "Booking confirmed", "Reference number:", "We've sent you...".
• Turns the user explicitly calls out as "this part has to be word-for-word."

Show the user:

"Turn 3's reply <short golden text> looks scripted. Pin it to exact-match (may_diverge: false) so any rewording breaks the test?"

Edit the JSON.

Workflow: A/B testing (prompt or model swap)

The killer use case. The user — not you — has decided to change the prompt, swap the model, or refactor a tool. You drive the test loop; you do NOT make the agent change yourself. See the HARD RULE.

# 1. Capture baseline BEFORE the change (or use the latest results.json).
trainforge run \
  --scenarios scenarios/*.json \
  --agent <agent spec> \
  --output baseline.json \
  --no-judge
# Label pending checks; trainforge rescore --results baseline.json.

# 2. THE USER makes the change (new prompt file, new model name in
#    config, etc.). You do not. Wait for them to confirm the change
#    is in place before running step 3.

# 3. Re-run with the override flag(s).
trainforge run \
  --scenarios scenarios/*.json \
  --agent <agent spec> \
  --output candidate.json \
  --no-judge \
  --override-model claude-sonnet-4-7
# Or: --override-prompt path/to/new-prompt.txt
# Label pending checks the same way; trainforge rescore.

# 4. Diff.
trainforge diff \
  --before baseline.json \
  --after candidate.json \
  --output regression.html

Then surface:

• Total regressed (was-passing-now-failing) scenarios — highest signal.
• Total fixed (was-failing-now-passing) scenarios.
• Which specific turn changed in each regression. Open the HTML report for visual diff if the user wants more detail.

--override-model X sets TRAINFORGE_OVERRIDE_MODEL=X env var AND ContextVar for the run. The user's agent must read the env / ContextVar inside the function (not at module load) for the override to take effect. If their agent caches model at import, an override won't fire — tell them to refactor or use a freshly-imported subprocess.

Workflow: failure debugging

When a test fails after a code change, don't just report it. Help the user understand why. Diagnose only. Never auto-fix. See the HARD RULE at the top of this file.

1. Read the failure. Open results.json. Find the failed scenario(s). Identify which turn / check broke.

2. Check recent code changes. Run git diff HEAD~1 (or whatever commit range matters). Look for changes in:

   • System prompts / instruction files
   • Model name in config
   • Tool definitions (signatures, names, return shapes)
   • Tool implementations (return values, error handling)
   • Agent function itself

3. Form a hypothesis. Examples:

   • Prompt file modified + scripted turn now fails exact-match → prompt regression.
   • Model name changed + multiple turns now produce different wording → model swap regression.
   • Tool function modified + tool arg failures appear → tool-impl change.
   • No recent code changes but test newly fails → environmental issue (env vars, API keys, external service).

4. Tell the user what you found. One sentence. Include the smoking-gun file:line if possible.

5. Recommend the next move — the user decides which one. Examples:

   • "The prompt change at prompts/system.txt:42 is the likely cause. Options: revert that change, or update the scenario's golden_response for turn 3 to match the new wording."
   • "Tool lookup_user now returns {user: {...}} instead of {...}. Options: revert the tool change, or update the scenario's expected_response for that tool to match the new shape."
   • "No recent code changes touched the agent path. Check OPENAI_API_KEY and any external service the agent depends on."

Strict no-go list for this workflow:

• ❌ Edit the user's prompt file to make the test pass.
• ❌ Change the model name in config to roll back a regression.
• ❌ Modify the tool implementation to match the scenario.
• ❌ "Suggest" a fix by silently applying it.
• ✅ Edit the SCENARIO JSON to record an intentional behavior change — but only after the user confirms the new behavior is desired.

If the user wants you to apply one of the fixes, that's fine — but they have to ask explicitly, and you should acknowledge the role switch out loud: "I'm stepping out of test-driver mode to edit your agent. Confirm?"

Workflow: consistency / reliability check

Use --runs N to measure agent flakiness. Useful when the user asks "is my agent reliable?" or "does this work every time?"

trainforge run \
  --scenarios scenarios/*.json \
  --agent <agent spec> \
  --output consistency.json \
  --no-judge \
  --runs 5
# Label pending checks; trainforge rescore.

Each scenario runs 5 times. The results JSON includes consistency: <pass_rate> per scenario and an inconsistent: true flag if pass rate < 80%.

Report to the user:

"Out of 5 runs per scenario, scenario X passed 3/5 (60%). That's below the 80% reliability threshold. Likely sources: model non-determinism on may_diverge: true turns, tool call ordering drift, or a real flakiness bug in the agent."

For agents with hot-path LLM calls (most agents), some non-determinism is normal. Flag scenarios where pass rate drops below 60% as real issues.

Workflow: visual HTML report

After every run + rescore, the user has results.json. For richer visual inspection (per-turn diffs, golden vs actual side-by-side, failure highlights, consistency summary), generate the HTML report:

trainforge report --results results.json --output report.html
open report.html

Offer this proactively after any rescore that produced failures or when the user asks "show me the details" / "what changed?" Don't generate one for every clean run — it's noise.

Workflow: multi-agent / node assertions

For agents that use sub-agents or a graph (LangGraph, OpenAI Agents SDK handoffs, custom orchestrators), scenarios can assert that specific sub-agents fired during a turn.

Detect when this applies: the user's agent code imports langgraph, crewai, uses OpenAI Agents SDK handoffs, or has explicit multi-agent orchestration. If so, offer:

"Your agent uses sub-agents. Want to add a node_assertion that verifies the refund_handler sub-agent fired on this turn — and didn't fire on simple greeting turns?"

To wire it up — note the split: the scenario JSON is yours to edit; the agent code is the user's:

1. The user's agent must wrap sub-agent calls in with trainforge.observer.node("name", args={...}): .... You do not write that wrap into their code. Show them the pattern and ask them to apply it. If they want you to do it, follow the HARD RULE: acknowledge the role switch and get explicit confirmation before touching agent code.
2. You write node_assertions: [{node_name: "refund_handler", must_fire: true}] into the scenario JSON (test data — yours).
3. Negative assertions: {node_name: "auto_approve", must_fire: false} for guard rails ("never auto-approve large refunds"). Also test data — yours.

Show the user the observer-wrap pattern (for them to apply):

from trainforge import observer

async def run_agent(messages):
    with observer.node("intent_classifier"):
        intent = classify(messages)
    if intent == "refund":
        with observer.node("refund_handler", args={"invoice_id": ...}):
            return await handle_refund(messages)

Important: node_assertions only fire under --agent (in-process) transport. Over --agent-url (HTTP), the runner has no way to read in-process state. If the user is HTTP-only, skip this workflow.

Workflow: pytest integration

When the user mentions CI, pytest, or wants tests in their test suite, drive the pytest plugin path.

pip install "trainforge[pytest]"

Configure in pyproject.toml:

[tool.pytest.ini_options]
trainforge_agent = "my_module:my_agent"
trainforge_scenarios_dir = "tests/agent/scenarios"

Then pytest tests/agent/ runs every scenario as one parametrized pytest case. Failures show the TrainForge diagnostic in the assertion message.

Limitations to flag to the user:

• One run per scenario (runs > 1 consistency scoring is CLI-only).
• In-process only (no --agent-url).
• No --no-judge mode in the pytest plugin yet; the user needs OPENAI_API_KEY + OPENAI_API_URL set for any scenario with LLM-judged checks. (When --no-judge lands in the plugin, this caveat will be removed.)

Workflow: CI setup snippet

When the user asks for CI integration, generate a GitHub Actions snippet adapted to their setup:

# .github/workflows/agent-tests.yml
name: Agent regression tests

on: [push, pull_request]

jobs:
  trainforge:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: '3.11'
      - run: pip install -e . trainforge
      - env:
          OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
          OPENAI_API_URL: ${{ secrets.OPENAI_API_URL }}
        run: |
          for f in scenarios/*.json; do
            trainforge run \
              --scenarios "$f" \
              --agent my_module:my_agent \
              --output "results-$(basename "$f" .json).json"
          done

CI runs use the configured-key path (drop --no-judge) since there's no coding agent in the CI loop to label pending checks. Document this trade-off clearly: friction-free coding-agent-as-judge is for local dev; CI needs a real LLM key.

For GitLab CI / CircleCI / etc., generate the equivalent. The shape is always the same: install, set OPENAI_API_KEY + OPENAI_API_URL, run the for-loop.

Default postures (Alex's opinion, baked in)

• ordered: false by default. Most tool loops in real agents are unordered. Force the user to opt in to ordered.
• may_diverge: false by default for the first turn the user reviews. This forces the user to decide which turns are scripted vs free-form, rather than defaulting everything to fuzzy.
• Every literal arg value in the transcript becomes expected unless the user marks it variable. The whole point is to catch silent regressions on arg values.
• One scenario per transcript by default. If the user wants to split, they'll say so.
• Write to the user's repo, not to a cloud. Tests are code. They live in git, version with the agent, and travel with the team.

What this skill DOES cover

Every feature shipped in this repo. Specifically:

• Capture: trainforge record --auto, paste-transcript fallback.
• Run: trainforge run with --no-judge (default) or configured LLM keys (CI path), --override-model, --override-prompt, --parallel, --runs N, --timeout.
• Label: per-check verdicts via the 20-NLP rubric (coding agent reads results.json, fills in pending checks).
• Score: trainforge rescore for deterministic re-aggregation.
• Diff: trainforge diff for before/after comparison (A/B testing workflow above).
• Report: trainforge report for HTML output (offer proactively on failures or when user asks).
• Standing rule: re-test on every agent change.
• Sharpening: post-capture outcome checks, tool arg constraints, scripted-turn pinning.
• Multi-agent: node_assertions for LangGraph / OpenAI Agents SDK / Anthropic Skills users.
• Failure debugging: read results, check git diff, form a hypothesis, suggest next move.
• Consistency: --runs N for flakiness measurement.
• Pytest integration: pip install "trainforge[pytest]" for test-suite users.
• CI: GitHub Actions / GitLab CI snippets on request.

What this skill does NOT do

The HARD RULE in one sentence: the skill is a test driver. The agent is the user's product. The skill never touches the product.

• It never edits the agent. No prompts, no model config, no tool definitions, no tool implementations, no agent function code. Even "obvious" fixes — never apply silently. Always diagnose + recommend; the user decides.
• It never auto-applies the "fix" for a failing test. If a test regresses because the prompt changed, the skill says "the prompt change is the cause; revert it OR update the scenario to match the new behavior." It doesn't pick.
• It does not invent tool calls the agent did not make. If the transcript doesn't show a tool call, the scenario doesn't assert one.
• It does not invent arguments. If the user says "the amount should be 950" but the transcript shows the agent calling with amount 1000, the scenario records what the agent actually did. Ask the user which one is the source of truth before deciding.
• It does not silently update scenarios when tests fail. On regression, surface the failure and let the user decide whether to update the scenario or revert the agent change.
• It does not invent LLM-judged verdicts. The coding agent labels pending checks using the rubric — applying the rule, not inventing one. If the rule is ambiguous on a specific case, fail the check with a brief explanation so the user can sharpen the scenario.
• It does not deploy infrastructure. No server, no Docker, no Kubernetes. The agent is just a Python function or HTTP endpoint; TrainForge calls it. Nothing to provision.
• It does not require API keys for the default flow. The --no-judge + coding-agent-as-judge path needs zero external credentials. Configured keys are only for CI runs.

The role-switch escape hatch

If the user explicitly wants you to fix their agent ("just fix the prompt", "update the model name for me"), step out of skill mode loudly:

"I'm stepping out of test-driver mode to modify your agent. The change I'd make is: <concrete change>. Confirm before I proceed?"

Wait for explicit confirmation. Then make the change. After the change, return to test-driver mode and re-run the scenarios. Do not silently cross the boundary.

This escape hatch is for users who explicitly opt in. It is NOT the default. If you're not sure whether the user is asking for a test-driver action or an agent edit, default to test-driver and ask.

Examples

See examples/transcript-restaurant-booking.md for a sample input transcript and examples/scenario-restaurant-booking.json for the scenario the skill should produce from it.