temper-skills

name

temper-skills

description

Give an agent skill's decision logic a test suite and a deterministic implementation — a labeled validation dataset plus versionable Python — via an adversarial multi-persona loop that runs natively in Claude Code: proposer plus persona subagents through the Task tool, entirely on your Claude Code subscription, no install and no API key. Use inside Claude Code when the user wants to "test", "eval", "audit", "temper", "distill", "freeze", or "harden" the routing/decision logic of a skill, prompt, or playbook ("test my skill", "what is this skill deciding?"), to audit a whole skills directory, or to turn a skill.md into a decision tree. Not for continuous scoring, text generation, or agents without a subagent primitive (there, use the temper-skills CLI/library instead).

Temper-Skills (subagent mode)

Turn the decision logic living in a prompt/skill.md into a deterministic Python decision tree — readable, diffable in a PR, zero LLM calls at inference. You (the orchestrator) act as the proposer; persona subagents are the adversarial reviewers. Everything runs on the user's Claude Code subscription via the Task tool.

"Adversarial" here means decision robustness — challenging business logic — not security scanning. The loop measures internal consistency, not correctness against the world; real correctness still comes from the user's ratified examples.

Operating rule — run autonomously (read this first)

You make the decisions; the loop is the point. The schema and the tree co-evolve — you start from a naive schema + a naive tree and let the loop grow and prune both. There are exactly two human touchpoints, and no others:

1. The per-round gate — Continue / Stop / Abort (skipped on quick). The only mid-run block.
2. The final review — before shipping, the user reviews the co-evolved schema, the outcome set, the recorded gray zones, and the validation dataset. This is review-the-output (a sign-off), not answering questions mid-run.

There is no up-front schema gate. Seed a naive schema yourself and start the loop; the schema_critic grows it, the outcome_critic flags a too-coarse outcome set, and the overengineering_critic + the earn-a-branch guard prune what doesn't earn its place. The schema is a loop output, reviewed at the end (it's the caller's integration contract), not a pre-commitment.

For everything else, decide and record — do not ask. Specifically, never raise an AskUserQuestion / multiple-choice modal to have the user choose feature breadth, whether to keep/drop a branch or feature, how to resolve a gray zone, or any "which design do you prefer" question. Those are the loop's job: the critics grow/prune, and you resolve gray zones with the safest defensible default and record them (see Gray zones). If you catch yourself about to open a question modal for anything other than the per-round gate: stop, make the defensible call, record it, and keep going.

Inputs

• A target skill.md / prompt (the logic to migrate), path given by the user. If the path is a directory (a skill library), do not start the loop: audit each discovered skill yourself — one judged pass per skill (decisiveness / combinatorics / stakes, plus how many features pin to a bounded value space) — and present a ranked findings table (skill · verdict · top finding · fix), most actionable first. The audit is a report worth having on its own; offer to temper the top pick, don't assume it.
• A schema: the pre-computed structured features the tree may branch on. If the user doesn't supply one, infer it from the skill (feature name + type + one-line meaning).
• Optional hard constraints (non-negotiable rules) and a few ratified examples.

Branch only on schema features. Conditions are valid Python boolean expressions over the feature names (e.g. food_item == "chocolate") and must be None-safe — any feature may be absent at inference, so guard before comparing (x is not None and x < 1, never bare x < 1) and coerce strings ((s or "").strip().lower()). A condition must never raise on a missing feature.

Seed a naive schema — the loop co-evolves it

Do not try to get the schema right up front, and do not gate on it. Seed a naive, tight schema — the features the source obviously decides on — and start the loop. Drop features that are circular (a feature that is the answer, like is_known_toxic_food); when in doubt leave a plausible feature out. Then let the loop co-evolve it:

• the schema_critic grows it — each round it can name a feature the source implies that the current schema can't express (e.g. minutes_since_exercise on decide_meal). You add it to the working schema so the very next round can branch on it.
• the earn-a-branch guard prunes it — a feature you added must earn a surviving branch within ~2 rounds (the proposer actually branches on it and it survives the overengineering_critic). If it never does, revert it out of the schema and record it as an advisory gap. This bounds schema size by tree size, so the loop still converges — the schema is no longer a fixed ceiling.
• the overengineering_critic prunes branches within the schema, every round.

The user does not choose feature breadth — that is exactly what the co-evolution is for. Offloading it to the user defeats the tool. The user's only schema involvement is the final review (below): they read the co-evolved schema + outcome set and sign off, because it's the contract the caller must extract against.

So the human touchpoints are exactly two: (1) the per-round Continue / Stop / Abort gate, and (2) the final review/sign-off. Nothing else is a question for them.

The personas

Four attackers plus three structural counterweights: the overengineering_critic (every round) and the two expressiveness critics — schema_critic and outcome_critic (both standard & audit-grade). Spawn one subagent per persona each round, in parallel (one message, multiple Task calls). Keep each subagent prompt lean and self-contained — do not make them read this skill or the repo.

The counterweights pull in three directions and are all essential; none adds validation cases:

• the overengineering_critic shrinks the tree within the current schema and outcome set;
• the schema_critic argues the input side is too thin — a feature the source implies but the schema can't express (e.g. on decide_meal, the missing minutes_since_exercise that forces the tree to punt);
• the outcome_critic argues the output side is too coarse — an answer the source implies but the outcome vocabulary can't express, so two distinct correct answers collapse into one label (e.g. on decide_meal, no way to say "wait, then a treat" — only wait_then_full_meal or treat_only). It's the exact dual of the schema_critic: challenge outputs, not just inputs.

Generated from temper_skills/sources.py — edit there, then run python -m temper_skills.skill_docs.

persona	always-on	angle (the style the model is given)
literalist	—	exploits literal ambiguities in the schema
edge_case_hunter	—	seeks rare combinations of feature values
bad_faith_actor	—	tries to strategically circumvent the rule
domain_expert	—	tests with rare but plausible domain cases
overengineering_critic	✅ every round	challenges every node: is this branch actually necessary, or is it loop richness rather than domain complexity?
schema_critic	✅ standard & audit-grade	argues the schema is too thin — names a feature the source implies that the schema cannot express, instead of adding test cases
outcome_critic	✅ standard & audit-grade	argues the outcome set is too coarse — names an outcome the source implies that the vocabulary cannot express (so two distinct answers collapse into one), instead of adding test cases

Each persona subagent returns ONLY this JSON:

{"persona": "<name>", "score": 0-10,
 "verdict": "ok|missing_case|collapsible|contradiction|schema_too_thin|outcome_too_coarse",
 "detail": "<one sentence>", "proposed_case": "<concrete feature assignment it mishandles, or null>",
 "proposed_tests": [{"input": {<feature: value>}, "expected": "<best-guess outcome>", "rationale": "<one line>"}],
 "proposed_features": ["<name: type — why the source needs it>"],
 "proposed_outcomes": ["<outcome — why (which two cases collapse today)>"]}

proposed_tests is how the validation set gets built (see below). Every attacker persona must add a case here whenever it finds a flaw — a full feature assignment plus the outcome it believes correct. Tell each subagent these are proposals a human will ratify, not ground truth. The three counterweights always return proposed_tests: []:

• The overengineering_critic removes complexity; it adds no cases, features, or outcomes.
• The schema_critic uses verdict: "schema_too_thin" and fills proposed_features — each a name: type — why for a feature the source implies but the schema can't express. You add load-bearing ones to the working schema (co-evolution, see below); the earn-a-branch guard reverts any that don't get used. They are growth signals, not branches you add to the tree.
• The outcome_critic uses verdict: "outcome_too_coarse" and fills proposed_outcomes — each an outcome — why for an answer the source implies but the outcome set can't express. Widen the outcome set when the gap is load-bearing; otherwise record it as a gray zone (the output-side dual). Every other persona leaves both proposed_features and proposed_outcomes empty.

score is always the TREE's robustness from your angle — 0 = the tree fails badly through your lens, 10 = solid, nothing to add. It is NOT how successful your attack was: a persona that finds no weakness scores the tree high (≈9–10) and returns verdict: "ok", proposed_case: null. Tell every persona subagent this explicitly so the scale is uniform across the panel.

Profiles & convergence

The profile sets the round budget, the panel, and how convergence is measured. These values are owned by temper_skills/distill.py — do not hand-edit the table:

Generated from temper_skills/distill.py — edit there, then run python -m temper_skills.skill_docs.

profile	max rounds	stop after N quiet rounds	per-round gate	provenance comments	adversary panel
quick	8	2	off	off	edge_case_hunter, overengineering_critic
standard	20	3	on	on	edge_case_hunter, domain_expert, schema_critic, outcome_critic, overengineering_critic
audit-grade	50	5	on	on	literalist, edge_case_hunter, bad_faith_actor, domain_expert, schema_critic, outcome_critic, overengineering_critic

Generated from temper_skills/distill.py — edit there, then run python -m temper_skills.skill_docs.

The loop stops when no round improves on the best for stop after N quiet rounds consecutive rounds (a plateau — whether high, a good tree, or low, can't improve), or the round cap is hit, or the user stops. The per-profile N and cap are in the profile table above. Convergence is a plateau, not an absolute score threshold.

The loop

1. Draft a naive initial tree yourself (proposer) from the skill + naive schema + constraints. Cover obvious cases. Where the source underdetermines an answer (a genuine gray zone — e.g. "the skill says 'when in doubt, say no' but never lists safe foods"), resolve it yourself with the safest defensible default consistent with the HARD constraints and the source's stated bias, and record it as a gray_zone on the node. Do not stop to ask the user how to resolve it — see "Gray zones" below.
2. Critique — spawn the profile's panel (the attackers + the structural counterweights) in parallel with the current tree; collect their scored JSON verdicts.
3. Co-evolve the schema/outcomes — apply the expressiveness critics before arbitrating so the same round can use them:
   • if the schema_critic named a feature (proposed_features), add it to the working schema and note the round you added it. Now the arbiter can branch on it.
   • if the outcome_critic named an outcome (proposed_outcomes), widen the outcome set if the gap is load-bearing (else record it as a gray zone).
   • earn-a-branch-or-revert: a feature/outcome you added must earn a surviving branch within ~2 rounds. If, ~2 rounds after you added it, no branch uses it (or the overengineering_critic would cut it), revert it and record it as an advisory gap. This keeps the schema no bigger than the tree needs.
4. Arbitrate — spawn one independent arbiter subagent, separate from you-the-proposer. It receives the current tree, the panel's verdicts, the constraints, and the current (possibly grown) schema — but not your defense of the draft — and rules kept / changed / rejected per persona with a one-line rationale (the arbitrage log), then returns the revised tree. It owes the draft no deference: keep a branch only because the logic and source justify it, not because it is already there; add a branch only if a critique justifies it AND an expert would write it by hand (including branching on a newly-added feature); collapse branches the overengineering_critic flags; never contradict a HARD constraint. You then apply its tree and track rounds_survived per node (matched by condition). Do not arbitrate your own draft — the proposer defending its own tree is the bias this split removes. (For a /unattended run you may fold the arbiter back into the proposer to save a spawn; standard/audit-grade always use a separate arbiter.)

Loop invariants

Generated from temper_skills/distill.py / validation_case.py — edit there, then run python -m temper_skills.skill_docs.

• earn-a-branch window: an added feature/outcome must earn a surviving branch within 2 rounds or be reverted to an advisory gap.
• case harvest excludes: overengineering_critic, schema_critic, outcome_critic — the structural critics restructure; they don't add cases.
• case statuses: proposed → resolved → ratified — only a human-set ratified gates anything.
• the only mid-run gate: Continue · Stop and review · Abort. Everything else is decided, recorded, and reviewed at the end.

Build the validation set as you go — written to disk every round

Pick one run_id for the whole temper (e.g. a UTC timestamp) and reuse it every round. Each round, after you apply the arbiter's tree, harvest the proposed_tests from every persona EXCEPT the structural critics — overengineering_critic, schema_critic, outcome_critic (they restructure; they don't add cases) — and write them to disk immediately by piping them to the deterministic per-round writer — do not wait for export:

echo '<this round's proposed_tests as a JSON list>' | \
  python scripts/update_validation.py <fn>.tree.json output/<fn>.py \
    --round <N> --run-id <run_id>

(scripts/ here is this skill's own vendored, stdlib-only copy — no install needed. The path is relative to the skill directory; use the absolute path to .../temper-skills/scripts/ if your working directory is elsewhere.)

This is not optional and not deferred: the committed output/<fn>.validation.jsonl and the behavior-lock test beside it must grow every round, so the user can watch the evidence accrue. The writer is deterministic (no LLM) and does the bookkeeping for you:

• dedups by input (the first round to find a case owns its label/status; a later duplicate only appends its source), so the same case found twice counts once;
• stamps first_seen_round + run_id on each new case — the audit trail that answers "was this generated this session?";
• refreshes every row's tree_prediction / agrees against the current tree — so when a round changes the tree, prior cases re-evaluate automatically. Run it even on a quiet round (pipe []) so a tree change still refreshes the dataset.

Each proposed_test is a {input, expected, rationale, source: "<persona>#r<N>"} object. Because you keep one <fn>.tree.json per decision on disk and re-export it each round (see the loop), the writer always scores against the round's live tree.

These are proposals, not ground truth. You are extending the validation set, not grading your own work: never treat a panel-authored label as a ratified anchor, and never let it gate anything. A disagreement ("agrees": false) is data to review, never a failing test — the behavior-lock test only asserts what the tree returns, and the ratified test (emitted only once a human blesses a label) is the only test allowed to fail. At review the user ratifies: they fix any wrong label and set "status": "ratified" — only then does it gate CI and anchor that cell on a re-run. Do not ask the user to ratify mid-run; it's a review-the-output step, like gray zones.

Gray zones are recorded, not interrogated

A gray zone is where the source genuinely underdetermines the answer. The plan resolves these by recording them and signing off at review (§2.5, §4.4) — they appear as # gray_zone: comments in the exported tree. So:

• Resolve each gray zone yourself with the safest defensible default (honor the source's bias — e.g. "when in doubt → no" means a conservative, minimal positive set), and record it as the node's gray_zone.
• Never raise a blocking question to resolve a gray zone, and never batch a list of design questions. The user ratifies or overrides by reading the recorded gray zones at the round gate or in the final output — that is the sign-off.
• The only blocking interaction in the whole run is the per-round Continue / Stop / Abort gate. If you're about to ask the user anything else, stop — make the defensible call, record it, and surface it at the final review.

Expressiveness gaps drive co-evolution (not a re-gate)

The schema_critic reports features the source needs but the schema can't express; the outcome_critic reports answers the source needs but the outcome set can't express. In the co-evolving model these are not a special "re-gate" — they are the normal growth signal:

• Grow, don't just record. When a critic names a load-bearing gap, add the feature to the working schema (or widen the outcome set) the same round, so the arbiter can branch on it next. You do not stop to ask the user — adding it is the loop's job.
• Earn-a-branch-or-revert bounds the growth. An added feature/outcome that never earns a surviving branch within ~2 rounds is reverted and recorded as an advisory gap (a gray_zone on the node that had to punt — e.g. an outcome gap "can't say 'wait, then a treat', so an already-fed dog that just exercised routes to treat_only"). This is what keeps a mutable schema from ballooning — a feature only survives if the tree actually uses it.
• The user sees it at the final review. The co-evolved schema, the outcome set, what grew, and what reverted are all surfaced for sign-off at the end — never as a mid-run question.

Export (deterministic — no LLM): emit a spec-compliant Agent Skill dir

When the loop ends, emit the tempered result as a spec-compliant Agent Skill folder (agentskills.io: SKILL.md + scripts/ + assets/) with the vendored, stdlib-only skill_render.py — no install. Write one <fn>.tree.json per decision, then a spec.json:

{"name": "route", "description": "...", "original_skill": "<path to original skill.md>",
 "generative_steps": ["<any step you left generative>"],
 "decisions": [{"tree": "route.tree.json", "module": "route",
                "schema": "route.schema.py", "consumes": []}]}

python scripts/skill_render.py spec.json output/route/

That writes the whole skill: output/route/SKILL.md (delegates each decision to its tree — extract features → call from scripts.<module> import <fn> → relay the verdict, never re-derive), scripts/<module>.py (the frozen tree, self-contained) + its behavior-lock and ratified tests, and assets/ (the schema + the .validation.jsonl dataset). One decision → a tempered skill; several → an orchestrator that chains them. It reconciles any existing .validation.jsonl and folds in the tree's proposed_examples, so the per-round dataset provenance (first_seen_round/run_id) is preserved.

(If you only want the raw tree + dataset, python scripts/export_tree.py tree.json route.py still does just that. The temper-skills package, when installed, offers the same via python -m temper_skills.export_skill, plus an LLM-woven variant that rewrites the original prose in its own voice.)

Show the user what changed: the original skill re-decided every call; the tempered skill extracts features, calls the tree, and relays the verdict — decision frozen, model still does NL extraction + phrasing (§2.5).

export_tree reconciles the committed route.validation.jsonl the per-round writer already built: it folds in any proposed_examples still in tree.json, refreshes every row's prediction against the final tree (without clobbering the first_seen_round/run_id provenance), and regenerates test_route.py (behavior lock — always green) plus test_route_ratified.py (only if a case is ratified — the sole test allowed to fail). Show the user the validation dataset as cases awaiting ratification, flagging every "agrees": false row — those are the highest-value disagreements to rule on. Debates live in the dataset as data, never as xfail tests.

export_skill is the deterministic template (default). If the user wants a woven variant that reads in the original skill's own voice, you may instead rewrite the original prose yourself — preserving its role/tone, deleting only the decision logic, and inserting the same delegation contract (extract features → call route.<fn> → relay verdict, never override; surface the recorded gray zones). Do not invent new rules; only re-route to the tree.

The tree JSON shape:

{
  "fn_name": "can_dog_eat",
  "features": ["food_item", "food_form", "dog_weight_kg"],
  "default_outcome": "unknown — verify manually",
  "model": "claude-<model> via claude-code-subagents",
  "profile": "standard",
  "nodes": [
    {"condition": "food_item == \"chocolate\"", "outcome": "toxic — never",
     "rounds_survived": 14, "sources": ["domain_expert", "constraints#1"],
     "gray_zone": null, "critic_note" 
  
   
        
      
  

proposed_examples is optional and proposals only — input / expected / rationale per case; the exporter adds the tree's prediction and "status": "proposed". In the normal loop you won't populate it: the per-round writer already accumulated the cases into route.validation.jsonl, and export reconciles that file. Use proposed_examples only for a one-shot export with no prior per-round dataset. Omit it if ratified examples already pin every contested cell.

The exported route.py carries a generated_at + model header (mandatory — a tree without a timestamp is not auditable), one inline provenance comment per node, and gray zones as trailing comments. It imports nothing and makes zero LLM calls at inference.

Complexity contract

The output must read like a tree a domain expert would hand-write. Tree depth/node count should reflect domain complexity, not how many rounds the loop ran. If you wouldn't accept the tree in a code review, it's over-engineered — let the overengineering_critic win that round. Provenance lives in comments, never in logic.