# Smelt
[](https://github.com/ProjAnvil/Smelt/actions/workflows/ci.yml)
**Smelt** โ a behavior-verification framework for agent skills. Put a skill in
the furnace: **given** contexts and an agent, **when** a trigger fires,
**then** assert on the outcome. One consistent yardstick to verify that each
version of a skill behaves better than the last.
Toolchain: uv + Python โฅ 3.11, zero core dependencies.
## Core concept: given / when / then
```python
from smelt import new_case, context, smelt_agent, text, tool_call, text_similar
result = (
new_case("commit-skill")
# given: contexts (fixture files, env vars, background prompts)
.given(context(files="fixtures/dirty_repo", prompt="the user is refactoring"))
# given: an agent (smelt's built-in LLM tool-loop agent; bring your own too)
.given(smelt_agent("skills/commit", llm=my_llm, tools=[git_tool]))
# when: a trigger โ a sentence, or a directory of files
.when(text("commit these changes for me"))
# then: assertions โ commands, JSON schema, strings, similarity to a reference
.then(tool_call("run_command", args={"cmd": "git status"}))
.then(text_similar("changes reviewed and committed", threshold=0.8))
.run()
)
result.assert_passed() # plugs straight into pytest
```
The fluent API is fully immutable (frozen dataclasses); baseline cases are safe
to derive:
```python
base = new_case("commit").given(smelt_agent(...))
case_a = base.when(text("commit this")).then(tool_call("git"))
case_b = base.when(text("leave it alone")).then(no_tool_call("git"))
```
### Alternative entry: agent-first + fragmented given
`smelt_agent` itself is an entry point; skill / llm / tools can be declared in
segments and are assembled at run time:
```python
from smelt import smelt_agent, skill, llm, tools
result = (
smelt_agent.new_case("commit")
.given(skill("skills/commit")) # or skill(prompt="You are...")
.given(llm(my_llm)) # any LLMClient; required
.given(tools(read_file, write_file)) # repeat to accumulate
.when(text("commit my changes"))
.then(tool_call("run_command"))
.run()
)
```
### Global default thresholds
`text_similar` and `llm_judge` thresholds have defaults (initially 0.8) that can
be tuned project-wide; an explicit threshold always wins:
```python
import smelt
smelt.configure(text_similar_threshold=0.7, llm_judge_threshold=0.75)
```
### .env and SMELT_* variables
Smelt reads credentials and endpoints from a `.env` file โ the location is
specified in code (default `.env` in the working directory):
```python
import smelt
smelt.configure(env_file="config/smelt.env") # or None to disable loading
smelt.load_env() # apply now; otherwise auto-loaded on first use
```
```dotenv
# config/smelt.env โ role-based quartet; SMELT_<ROLE>_<KEY> wins, legacy
# shared keys (SMELT_API_KEY / SMELT_BASE_URL / SMELT_LLM_PROVIDER) are the
# common fallback. Roles: JUDGE, AGENT, DOCTOR, CHALLENGER.
SMELT_JUDGE_PROVIDER=openai
SMELT_JUDGE_BASE_URL=https://api.moonshot.cn/v1
SMELT_JUDGE_API_KEY=sk-...
SMELT_JUDGE_MODEL=kimi-k2
SMELT_DOCTOR_PROVIDER=anthropic
SMELT_DOCTOR_API_KEY=sk-ant-...
SMELT_DOCTOR_MODEL=claude-sonnet-4-5
```
All variables carry the `SMELT_` prefix. `OpenAIChatClient` falls back to
`SMELT_API_KEY` / `SMELT_BASE_URL` when the arguments are omitted (explicit
arguments win), and `smelt evaluate` falls back to `SMELT_JUDGE_MODEL` when
`--judge-model` is absent. Variables already present in the shell environment
are never overridden by the file.
## given
| helper | description |
|---|---|
| `context(files=..., prompt=..., env=..., vars=...)` | fixture files/dirs copied into an isolated workspace; env injected into tool execution; prompt appended to the system prompt; stackable and merged |
| `reference(path)` / `reference_folder(dir)` | mount skill resources (references/ scripts/ assets/) into the workspace **and register them** โ declaration doubles as registration for coverage reporting |
| `smelt_agent(skill, llm=..., tools=[...])` | built-in agent: loads SKILL.md as the system prompt, runs the "LLM โ tool โ feedback" loop |
| `subagent(agent_md, llm=..., tools=[...])` | run a Claude agent .md as the system under test โ body is the system prompt, tools/maxTurns frontmatter enforced |
| `router_agent(agents_dir, llm=...)` | the simulated main agent: sibling roster prompt + Task tool stub, for delegation-routing cases |
| `fixed_agent(output, tool_calls=[...])` | fixed-output agent: replays a preconfigured trace without an LLM โ for assertion self-tests and baseline regression |
| `skill_catalog(source, distractors=..., gold=..., k=5)` | fixed candidate pool for discovery cases; renders `name` + `description` only, never the SKILL.md body |
| `discovery_agent(catalog, llm=...)` | simulated main agent that selects an ordered skill list via the `select_skill` tool (level 1 discovery) |
| custom | implement the `Agent` protocol (`run(ctx, input) -> Trace`) to plug into every assertion |
## when
| helper | description |
|---|---|
| `text("...")` | a one-sentence trigger, delivered as the user message |
| `task("...")` | a delegation-shaped trigger (labeled "task"; what a main agent would send) |
| `directory("path/")` | a directory trigger: contents materialized into the workspace, file listing rendered into the user message |
## then (each assertion scores independently with its own threshold)
| helper | scoring |
|---|---|
| `tool_call(name, args={...})` | exact argument-subset match 1.0; name only 0.5; never called 0 |
| `no_tool_call(name)` | negative assertion |
| `output_equals(s)` / `output_contains(*s)` | exact / per-fragment ratio |
| `text_similar(reference, threshold=0.8, scorer=...)` | similarity to the reference; difflib by default, inject an embedding scorer if you like |
| `json_output(schema, contains={...})` | parseable 0.4 + schema 0.4 + field subset 0.2; full JSON Schema with `smelt[json]` installed |
| `llm_judge(judge, criteria=..., reference=..., threshold=0.8)` | LLM-as-judge: a judge model scores the output 0~1 against a rubric |
| `turns_used(max=N)` / `tool_budget(name, max=N)` / `wall_time(max_seconds=S)` | budget gates: agent loop turns / tool call count / wall-clock time โ deterministic, binary |
| `delegates_to(name)` / `no_delegation()` / `no_delegation_to(name)` | routing assertions on the Task tool's `subagent_type`: right target 1.0, wrong target 0, no delegation 0/1 |
| `reference_read(path)` / `no_reference_read(path)` | reference reach / restraint โ tool-name-agnostic path matching (absolute and `../` spellings detected) |
| `reference_untouched(path, source=...)` | ablation validity: proves via content fingerprint that the reference never entered the context |
| `skill_selected(name, k=1)` / `skill_rank(name, max_rank=1)` | the selected skill appears within the first `k` / at or above a 1-based rank |
| `no_skill_selected()` / `skill_not_selected(name)` | NONE-set and near-miss negatives on the `select_skill` list |
| `skill_recall(expected, k=5)` | graded `\|expected โฉ top-k\| / \|expected\|` |
### Testing skill discovery (progressive disclosure level 1)
A skill can behave perfectly once loaded and still never be found. Level 1
discovery tests the trigger step: the agent sees only `name` + `description`
from a fixed catalog and returns an **ordered** `select_skill` list. An empty
list is a valid answer, so NONE queries can measure false triggers.
```python
from smelt import (
discovery_agent, discovery_query, discovery_suite, new_case,
no_skill_selected, skill_catalog, skill_selected, text,
)
catalog = skill_catalog(
"examples/discovery/skills",
distractors="examples/discovery/skills/_distractors",
k=5,
)
hit = (
new_case("pdf-hit")
.given(discovery_agent(catalog, llm=router_llm))
.when(text("pull the tables out of this scanned PDF into a spreadsheet"))
.then(skill_selected("pdf-table-extract"))
)
report = discovery_suite(
catalog=catalog,
queries=[
discovery_query("pull the tables out of this scanned PDF into a spreadsheet",
gold=["pdf-table-extract"], kind="positive"),
discovery_query("forward this PDF to the client as an attachment", kind="near-miss"),
discovery_query("tell me a joke", kind="none"),
],
llm=router_llm,
repeat=3,
).run()
report.assert_metrics(hit_at_1=0.90, false_trigger_rate=0.10)
```
The report exposes `hit_at_1`, `recall_at_1`, `recall_at_k`, `mrr_at_10` and
`false_trigger_rate`, broken down by query `kind` and by surface-overlap bucket.
Low-overlap Hit@1 is the headline number: a query that merely repeats the
description words is easy and hides keyword stuffing. `train`/`test` splits keep
description tuning off the held-out set. Discovery cases are ordinary cases, so
`repeat`, `pass^k`, `compare()` and the coverage gates apply unchanged.
`DiscoveryReport.to_html()` renders a standalone page; `evaluate_skill(...).save("report.html")`
includes the same Discovery section in the evaluation report.
```bash
uv run smelt discover examples/discovery/skills \
--distractors examples/discovery/skills/_distractors \
--queries examples/discovery/queries.py --repeat 1 \
--min-hit-at-1 0.8 --max-false-trigger-rate 0.2 \
--output .smelt/discovery-report.json --fail-under 80
```
`evaluate_skill(...).with_discovery(...)` attaches the same suite to the
comprehensive report. The `discovery` part joins `overall_score` with a default
weight of `0.2`. Because the weighted total still counts only parts with data,
evaluations without a discovery suite keep their historical scores exactly.
`compare()` reports discovery metric deltas and per-query verdicts; when only
one side has a discovery section it emits a scoring-basis-change note instead of
a silent numeric comparison. See `examples/discovery/README.md` for the query
set discipline and hard-distractor provenance.
### Testing references (progressive disclosure level 3)
Skills with depth in `references/` need three distinct probes:
```python
# reach โ the task needs it, so the agent must consult it
.then(reference_read("references/endpoints.md"))
# restraint โ not needed here, so it stays unread
.then(no_reference_read("references/performance.md"))
# ablation validity โ in a WITHOUT arm, prove the content never leaked in
.then(reference_untouched("references/endpoints.md", source="skills/reddit"))
```
**With/without ablation** (measuring what a reference contributes): the without
arm simply does not mount the reference โ never mount an empty file instead
(empty โ absent; it confounds the measurement). For the cleanest counterfactual,
also drop the pointer sentence from the skill prompt via `skill(prompt=...)`.
Diff the two arms with `compare()` to get the lift. Content-level mocks for
robustness testing use `mock_tool(read_file, {"references/x.md": ""})`.
`evaluate_skill` reports a **Reference Coverage** section: every referenced
file (registered by `reference*` declarations โช scanned from SKILL.md) marked
reached/unreached across all behavior cases; `compare()` surfaces
reachedโunreached transitions between versions.
### Budget assertions (operating envelopes)
Quality can pass while the run is unaffordable โ the "budget burner" failure
mode. Budget assertions are deterministic gates on the trace (no LLM call),
catching versions that get slower or more expensive:
```python
.then(turns_used(max=5)) # 3 turns vs 8 turns no longer score alike
.then(tool_budget("run_command", max=2)) # or tool_budget(max=4) for the total
.then(wall_time(max_seconds=10))
```
The runner times every run (`trace.wall_time_s`) and agents report their turn
count (`trace.turns`); when the LLM client reports token usage it is summed
into `trace.metadata["token_usage"]`. Because budgets are ordinary
expectations, they flow into repeated sampling, compare(), and CI gates like
any other assertion.
Case score = mean of assertion scores; an assertion passes when `score >= threshold`.
Custom assertions implement the `Expectation` protocol (`evaluate(trace) -> ExpectationResult`).
## Repeated sampling (mean ยฑ std)
LLM agents are stochastic โ a single run's score is noise, and noise can be
larger than the difference between two skill versions. Repeat a case and the
score becomes the mean across runs, with the spread (std) reported alongside:
```python
result = (
new_case("commit")
.given(smelt_agent("skills/commit", llm=real_llm, tools=[git_tool]))
.when(text("commit my changes"))
.then(tool_call("run_command"))
.repeat(3) # or: .run(times=3)
.run()
)
result.score # mean of 3 run scores (a crashed run counts as 0)
result.score_std # spread โ large std means "result unstable, don't compare"
result.run_scores # per-run scores, e.g. [1.0, 0.5, 1.0]
result.pass_hat # pass^k reliability: True only if EVERY run passed
```
The mean flatters an agent that succeeds sometimes; pass^k (tau-bench's
reliability metric) answers "can this skill be trusted every single time".
Both are shown in reports: `0.67 ยฑ0.47 (n=3) pass^3 โ` means "usually works,
but not reliably" โ a signal the mean alone would hide.
Aggregation: case score = mean of per-run case scores; each expectation's
score = mean across the runs that evaluated it; pass = mean >= threshold.
Configuration errors (missing agent/trigger) never repeat; runtime errors
count as 0 and are listed as run errors. HTML / terminal / markdown reports
all show `ยฑstd ยท n=N` next to repeated scores.
## LLM clients
```python
from smelt import ScriptedLLM, LLMResponse, OpenAIChatClient
# Deterministic replay (CI default): emit canned responses in order
llm = ScriptedLLM([
LLMResponse.call("run_command", {"cmd": "git status"}),
LLMResponse.say("status checked"),
])
# Real models (regression / comparing skill versions): OpenAI-compatible API
llm = OpenAIChatClient(model="kimi-k2", base_url="https://api.moonshot.cn/v1")
```
## LLM-as-judge
`llm_judge` lets another model grade the output: a judge model scores 0~1
against a rubric or reference answer, passing at the threshold. The judge is any
`LLMClient` and may differ from the agent under test (e.g. a stronger model):
```python
from smelt import llm_judge, new_case, smelt_agent, text
result = (
new_case("refusal-quality")
.given(smelt_agent("skills/refund", llm=agent_llm, tools=[...]))
.when(text("I refuse to return it, just give me the money"))
.then(llm_judge(
judge_llm,
criteria="must politely decline the non-compliant request and offer a compliant alternative",
threshold=0.7,
include_trace=True, # hand the tool-call process to the judge too
))
.run()
)
```
The judge sees the task input (the when-trigger) by default โ grading an
answer without seeing the question inflates scores and blurs version
differences; `include_input=False` opts out. With `include_trace=True` the
judge sees tool calls **including their results**, so it can tell whether the
agent actually used a tool's output or hallucinated from memory. Keep the
judge at temperature 0 (`OpenAIChatClient`'s default), and preferably a
different model family than the agent under test (judges rate their own
family's output higher).
### Per-dimension judging
For a fuller picture of *where* a skill is weak, grade independent dimensions
โ each in a **separate** judge call on a categorical 0|1|2 (fail/partial/pass)
scale, averaged into the expectation score:
```python
.then(llm_judge(
judge_llm,
dimensions=[
"tool selection (right tool, or correctly none)",
"argument correctness (arguments semantically match the request)",
"result utilization (the answer actually uses what tools returned)",
"trajectory efficiency (no loops or redundant calls)",
"error recovery (recovers sensibly from tool errors)",
],
threshold=0.7,
include_trace=True, # dimensions like result utilization need the tool results
))
```
One call per dimension prevents anchor bleed (a strong first dimension
dragging the others up) and makes regressions attributable โ the result's
`details["dimensions"]` carries each dimension's level and evidence-based
reason. Scoring rationale precedes the score in every judge prompt
(reason-first judging), and all prompts state length neutrality explicitly.
Judge parse failures, out-of-range scores, and call failures all score 0 safely
with an explanatory message โ they never crash the case.
## langchain integration (optional)
The core stays dependency-free. With existing langchain model configs, install
`smelt[langchain]` and bridge via `LangChainLLM` (langgraph intentionally unused โ
judge and agent both need a single completion, not a graph runtime):
```python
from langchain_openai import ChatOpenAI
from smelt.given.agents.langchain_llm import LangChainLLM
llm = LangChainLLM(ChatOpenAI(model="kimi-k2", base_url="https://api.moonshot.cn/v1"))
# identical to ScriptedLLM / OpenAIChatClient: feed smelt_agent, or use as judge
```
### Provider factory (openai / anthropic, base_url mode)
Or let smelt construct the chat model โ provider mode, optional base_url
(provider default when omitted), api_key; three config keys in your `.env`:
```dotenv
SMELT_LLM_PROVIDER=anthropic # or openai (default)
SMELT_BASE_URL=https://proxy... # optional; omitted โ provider default endpoint
SMELT_API_KEY=sk-...
```
```python
llm = LangChainLLM.from_env("claude-sonnet-4-5")
# or explicitly (explicit args win over the env file):
llm = LangChainLLM.from_provider("openai", "kimi-k2", base_url="https://api.moonshot.cn/v1")
```
Requires `smelt[langchain-openai]` or `smelt[langchain-anthropic]`.
## Comprehensive evaluation & reports (evaluate)
Produce a full evaluation report for a skill: **behavior score** (then-case
results) + **writing score** (per-dimension LLM review of SKILL.md) + **lint
score** (static checks), weighted into an overall grade (AโF), with improvement
suggestions generated code-first from lint findings (the judge LLM only adds
semantic suggestions on top). Fully code-driven:
```python
from smelt import evaluate_skill, new_case, text, tool_call
report = (
evaluate_skill("skills/commit", judge=judge_llm, agent_llm=agent_llm, tools=[git_tool])
.with_cases(
# unbound cases auto-bind the skill under review โ replay one suite across versions
new_case("commit").when(text("commit my changes")).then(tool_call("run_command")),
)
.with_lint() # static lint part (on by default)
.with_writing(dimensions=[...]) # LLM writing review, custom dimensions (on by default)
.with_suggestions(max_items=5) # code-first checklist suggestions (LLM adds semantic ones)
.with_weights(behavior=0.5, writing=0.3, lint=0.2) # adjustable weights
.with_times(3) # repeat count for behavior cases (see below)
.run()
)
print(report.overall_score, report.grade) # 89.5 'B'
report.save("reports/commit.md") # extension picks the format; .json โ JSON, .html โ HTML
report.to_html() # self-contained HTML page, includes the Discovery section
```
**Behavior cases repeat automatically.** Without an explicit setting, each
case runs 3 times and scores aggregate as mean ยฑ std โ that is what makes two
skill versions comparable. Cases driven by deterministic backends
(`fixed_agent`, `ScriptedLLM`) auto-degrade to a single run, so CI replay
stays cheap. Precedence: `.with_times(n)` > per-case `.repeat(n)` > auto.
The markdown/JSON reports carry `runs`, `run_scores` and `score_std` per case.
Report sections: overview (overall/grade/parts) โ behavior details (per-then
score vs threshold) โ static lint table โ writing review table (dimension /
score / comment) โ prioritized suggestions. Without a judge, the writing review
is skipped and suggestions are derived from lint findings alone, so a checklist
report is always produced.
### Comparing skill versions (compare)
`compare()` diffs two evaluations โ SkillEvaluation objects, payload dicts, or
saved `.json` reports โ and tells you whether the numbers *mean* something:
```python
from smelt import compare
diff = compare("reports/skill-v1.json", "reports/skill-v2.json")
print(diff.to_markdown()) # per-case table: baseline / candidate / ฮ / verdict
diff.assert_no_regression() # CI gate: raises on any regressed case
```
A case's score change is **significant only when |ฮ| exceeds the noise band**
`max(min_delta, 2ฯ)` โ ฯ combines both versions' per-run spreads, so a 3-point
"bump" inside the band is reported as `unchanged`, not progress. Independently
of the mean, a **pass^k drop (True โ False) is flagged as a reliability
regression** โ the version still scores fine on average but no longer works
every time. Verdicts: `improved` / `regressed` / `unchanged` / `added` /
`removed`; a candidate-side runtime error on an existing case is a regression.
CLI equivalent (exit 1 on regression โ drop straight into CI):
```bash
smelt evaluate skills/commit --cases cases.py --output reports/v1.json ...
# ...edit the skill...
smelt evaluate skills/commit --cases cases.py --output reports/v2.json ...
smelt compare reports/v1.json reports/v2.json --min-delta 0.05
```
CLI equivalent:
```bash
smelt evaluate skills/commit --cases cases.py \
--judge-model kimi-k2 --base-url https://api.moonshot.cn/v1 \
--output reports/commit.md --fail-under 80
```
### Adversarial challenge (on by default)
Every `evaluate` run also sends a challenger agent after the skill: it reads
SKILL.md and crafts probes โ awkward phrasings that should trigger, near-miss
requests that must not, and distractor fixtures with misleading content.
Breaks are reported with a suggested `new_case(...)` snippet for human review.
```python
evaluate_skill("skills/commit", judge=judge_llm, challenger=red_llm)
.with_challenge(rounds=2, probes=8) # defaults: on, 1 round, 8 probes
# .with_challenge(enabled=False) # explicit opt-out
```
Challenge is **advisory**: it never enters `overall_score` (probes are
regenerated each run โ scoring them would make the same skill version measure
differently twice). `compare()` shows a breaks-count diff informationally and
never gates on it. To opt a challenge score into the total anyway:
`with_weights(behavior=0.4, writing=0.2, lint=0.2, challenge=0.2)`.
The challenger resolves from `SMELT_CHALLENGER_*` (role quartet); prefer a
different model family than the agent under test โ same-family challengers
share blind spots. Missing challenger config degrades the section to
"skipped"; other parts still score.
## Evaluating Claude agent files (.claude/agents)
Claude Code subagent definitions โ markdown files with `name` / `description`
/ `tools` / `model` frontmatter โ are a first-class document kind across smelt:
```python
from smelt import (
new_case, text, task, tool_call, no_tool_call,
evaluate_agent, subagent, router_agent,
delegates_to, no_delegation, no_delegation_to,
)
# Delegation routing: does the main agent pick the right subagent?
(
new_case("route-release-notes")
.given(router_agent(".claude/agents", llm=router_llm)) # roster prompt + Task tool
.when(text("draft the v2.0 release notes"))
.then(delegates_to("release-notes"))
)
(
new_case("no-route-trivial")
.given(router_agent(".claude/agents", llm=router_llm))
.when(text("rename this file to foo.py"))
.then(no_delegation()) # near-miss tasks must not be delegated
)
# Agent body: the markdown body is the system prompt; frontmatter applies
(
new_case("body-no-rogue-write")
.given(subagent(".claude/agents/release-notes.md", llm=agent_llm))
.when(task("Summarize PR #142 for the changelog")) # simulated delegation message
.then(tool_call("git_log"))
.then(no_tool_call("write_file")) # tools allowlist is enforced
)
# Full report โ same shape as evaluate_skill, AโF grade
report = (
evaluate_agent(".claude/agents/release-notes.md",
judge=judge_llm, agent_llm=agent_llm, tools=[git_log])
.with_cases(...)
.run()
)
```
- `smelt validate .claude/agents` lints the official Claude Code load contract
(skip rules, unknown fields with camelCase hints โ Claude Code silently
ignores typos, tool patterns incl. `mcp__server__*` / `Agent(name)`, model
validity, maxTurns) plus delegation-oriented description checks.
- Unbound cases auto-bind by shape: `delegates_to` / `no_delegation` cases bind
a router over the file's directory; everything else binds a subagent on the
file. Frontmatter is enforced โ the tools allowlist filters provided tools
(an allowlist that empties a non-empty tool set fails the run, mirroring
Claude Code's launch refusal), `maxTurns` caps the loop, `model` lands in
trace metadata.
- `smelt doctor cases.py --agent .claude/agents/release-notes.md` mutates
descriptions (drop one sentence / make it generic) and sections โ routing
cases guard the description, body cases guard the sections.
- `evaluate`'s challenger probes routing: near-miss tasks that must not
delegate (`no-route`), awkward phrasings that must (`route`), and genuinely
unclear ones (`ambiguous`, judged).
## Health-checking the yardstick (smelt doctor)
Cases written by an agent can be false-green: they pass whether the skill is
good or bad. `smelt doctor` questions the yardstick itself:
```bash
smelt doctor cases.py --skill skills/commit --min-score 0.8
```
- **Mutation check** โ deliberately breaks the skill (drops one section /
reference / constraint at a time) and re-runs the cases. A case suite that
stays green against a broken skill is blind; the report lists surviving
mutants with a suggested case each. Mutation score = killed / scorable
mutants; kill verdicts use the same noise band as `compare()`.
- **Judge canary** โ feeds the judge one deliberately bad trace (off-topic +
hallucinated tool result); a calibrated judge scores it below 0.5.
- **`@mutate_check(guards="section:Boundaries")`** โ declare which skill part
a case guards; doctor verifies the case actually kills that mutant, and
flags false or dangling guard claims.
Exit codes: 0 healthy ยท 1 issues found ยท 2 configuration error. The doctor
agent is configured independently (`SMELT_DOCTOR_*`); if it is missing,
doctor refuses to run (silent skipping would be false green).
## Per-case reports (.report / .report_cli)
Every case chain can end in a report:
```python
result = (
new_case("commit")
.given(smelt_agent(...))
.when(text("commit my changes"))
.then(tool_call("run_command"))
.report() # writes a self-contained HTML report to .smelt/reports/
)
result.report_path # ".smelt/reports/commit-20261006-091500.html" (+ commit-latest.html)
result.assert_passed() # still pytest-compatible
# or straight to the terminal:
new_case("commit").given(...).when(...).then(...).report_cli()
```
- `.report(output_dir=..., quiet=...)` โ dark, self-contained HTML: status badge,
per-expectation score bars, tool-call trace, final output; a `<slug>-latest.html`
pointer always tracks the newest run
- `.report_cli(quiet=...)` โ formatted terminal report with score bars and the
tool-call trace
- Invalid cases (missing agent/trigger) produce an error report instead of raising
## Suite reports (multi-case overview)
Run a whole battery and get an `index.html` overview โ pass rate, score
distribution, per-case pages linked:
```python
from smelt import suite
result = (
suite("commit-skill v1.2")
.add(case_a, case_b, case_c)
.report() # .smelt/reports/suites/commit-skill-v1-2-<ts>/index.html + one page per case
)
result.assert_passed()
suite("commit-skill v1.2").add(...).report_cli() # compact terminal summary
```
Layout: `suites/<slug>-<timestamp>/index.html` with `<case>.html` siblings, plus a
`suites/<slug>-latest/` copy that always tracks the newest run โ history is kept
per timestamp, ideal as a CI artifact.
## Running
```bash
uv sync --extra dev
# pytest (recommended; the fixture self-registers)
uv run pytest
# CLI: execute case files directly (module-level new_case(...) or cases = [...])
uv run smelt run examples/cases/demo_cases.py -v
# budget assertions demo: efficient vs wasteful agents (the wasteful case fails on purpose)
uv run smelt run examples/cases/budget_cases.py -v
# static lint: structure / trigger words / asset references for SKILL.md
uv run smelt validate examples --fail-under 80
# L1 discovery / recall suite (deterministic demo; use --repeat 3 with a real model)
uv run smelt discover examples/discovery/skills \
--distractors examples/discovery/skills/_distractors \
--queries examples/discovery/queries.py --repeat 1
```
Two assertion styles under pytest:
```python
def test_commit():
new_case("commit").given(...).when(...).then(...).run().assert_passed()
def test_commit_fixture(smelt): # the smelt fixture comes from the pytest plugin
smelt.check(new_case("commit").given(...).when(...).then(...))
```
## Runner guardrails
| shape | behavior |
|---|---|
| no given (no agent/fragments), straight to when/then | โ fails with "missing agent" |
| given with context only (no agent) | โ fails with "missing agent" |
| fragmented given without llm | โ fails with "missing llm fragment" |
| given without when | โ fails with "missing trigger" |
| given + when, no then | โ runs; empty assertions pass vacuously (flow check) |
| agent crashes at run time | โ error converges into `CaseResult.error` |
| double agent / double when / wrong types | build-time error, raised immediately |
## Project layout
```
src/smelt/
โโโ given/ # the precondition side
โ โโโ context.py # context(): fixtures/env/prompt โ isolated workspace
โ โโโ fragments.py # skill()/llm()/tools() fragments and assembly
โ โโโ agents/ # Agent protocol, SmeltAgent, FixedAgent, LLMClient, LangChainLLM
โโโ when/ # the trigger side: text / directory
โโโ then/ # the assertion side: tool_call / text_similar / json_output / llm_judge / ...
โโโ discovery/ # L1 skill discovery: catalog, labelled queries, suite + IR metrics
โโโ case.py # SmeltCase: fluent immutable given/when/then
โโโ runner.py # guardrails โ materialize context โ run agent โ evaluate
โโโ report/ # reports: per-case HTML/text + suite overview (index.html)
โโโ evaluate.py # evaluate_skill: behavior + writing + lint review and reports
โโโ config.py # global default thresholds (smelt.configure)
โโโ tools.py # @tool: signature annotations โ JSON Schema
โโโ pytest_plugin.py # the smelt fixture
โโโ cli.py # smelt run / validate / evaluate / discover
โโโ lint/ # static quality lint submodule (former skillcheck)
```
## License
MIT