| name | tau2-playbook |
| description | τ²-bench-specific benchmark guidance (Sierra Research; retail / airline / telecom). Multi-turn dialogue eval with multiplicative reward (DB × action × NL-assertion) and pass^k reliability metric. Catalogues tau2 failure modes and techniques from public writeups, mapped to HarnessX's four levers (config / control / action / instruction). Use when forming a hypothesis or scanning for which intervention fits a pattern seen in tau2 trajectories. |
τ²-bench: Benchmark-specific Guidance
Current Benchmark
-
What: τ²-bench (Barres et al., Sierra, arXiv 2506.07982) — multi-turn
dialogue eval where an LLM-driven user converses with the agent, the
agent calls domain tools that mutate a shared database, and a
three-component evaluator scores the run. Evolves out of τ-bench
(ICLR'25, retail+airline) → τ²-bench (adds telecom as dual-control
Dec-POMDP) → τ³-bench (Oct'25, adds banking+voice). Repo:
github.com/sierra-research/tau2-bench. HarnessX targets τ².
-
Domains used by this evolver:
| Domain | Tasks (HarnessX split) | Agent tools | Grading |
|---|
retail | 114 | 7 W + 6 R | DB-state equality (strict) |
airline | 50 | 6 W + 6 R | DB-state equality (strict) |
telecom | 114 (base split) | 6 W + 7 R + user-side tools | outcome-state (lenient) |
Full telecom pool is 2285 tasks; the runner always uses base.
-
Reward ∈ [0, 1] = product of three components; any zero zeros everything:
- DB check — final database matches ground truth.
- Action checks — per-action
action_match booleans.
- NL assertions — LLM judge rates claims about the dialogue (temp 0.3).
-
pass^k vs pass@k — Sierra's signature framing. pass@k = at least
one of k runs passes; pass^k = all k runs pass. A 70 % pass^1
agent collapses to ~25 % pass^8. We currently run pass^1
(defaults.NUM_TRIALS = 1); raise NUM_TRIALS to surface reliability.
-
Realistic baselines (τ²-verified avg@3, Nova-2 report Mar 2026 —
citable public numbers). Airline is uniformly hardest (brittle
single-solution grading); telecom is highest (multiple valid paths):
| Model | Telecom | Retail | Airline |
|---|
| GPT-5 | 86.5 | 78.3 | 72.0 |
| Claude Sonnet 4.5 | 78.1 | 77.2 | 66.8 |
| GPT-5-mini | 71.1 | 73.7 | 68.8 |
| Claude Haiku 4.5 | 54.7 | 69.1 | 54.0 |
HarnessX's own runs are in benchmarks/tau2/README.md.
Task Structure
Each task is a multi-turn dialogue built around one customer-service
goal: user opens with an ambiguous request → agent info-gathers via read
tools → diagnoses and proposes → user confirms → agent executes write
tools → user acknowledges or adds a follow-up, then stops (###STOP###).
Tools split into two classes by name convention (basis of
PhaseAwareToolFilter in benchmarks/tau2/tool_filter.py):
| Class | Prefixes | Examples |
|---|
| Read | get_, list_, search_, find_, check_, look_, think* | get_order_details, list_orders |
| Action | everything else | refuel_data, enable_roaming, toggle_* |
Per-Domain Cheat-Sheets
Cross-domain guidance is usually wrong in subtle ways. Anchor any
instruction-layer fix to the specific domain that surfaced the failure.
retail — order status drives tool choice: modify_pending_order_items
and cancel_pending_order need pending; exchange_delivered_order_items
needs delivered. Returns must use the original payment method (no
swapping). Exchanges must be same product, different option. Courtesy
credits often require escalation, not silent issue.
airline (hardest by pass^1) — basic-economy cannot be modified
(only cancellable via insurance); strict baggage rules per cabin; user
ID and reservation ID must be elicited, never fabricated. Hidden user
constraints ("I only accept EWR, not JFK") live in user_instruction
and surface only when the agent asks. Single-solution ground truth means
reward is brittle — a correct alternative can still score zero.
telecom (dual-control) — task-class hierarchy: service_issue <
mobile_data_issue < mms_issue; harder classes require resolving
underlying issues first. The agent cannot toggle user-side settings
(airplane mode, cellular data, roaming toggle, APN); it must instruct
the user in natural language, wait for confirmation, then re-read
state. PolicyHintProcessor (benchmarks/tau2/policy_hint.py) is an
off-by-default telecom companion that injects positive remediation
hints when conditions like data_exceeded, roaming_disabled_abroad,
check_correct_line are detected in tool results.
Analyzing Failures
Frontmatter fields emitted by the evolver (always present):
| Field | What it tells you |
|---|
task_id | τ² task identifier |
exit_reason | done / max_steps / max_errors / too_many_errors / agent_stop |
reward | Float [0,1]; 1.0 = full pass |
eval_passed / eval_score / eval_reason | SOUL.md-generic aliases for reward >= 1.0, reward, and the termination_reason + failed-action names |
num_messages | Total messages exchanged |
cost_usd / elapsed_s | Cost + wall clock |
tools_used / tool_call_counts / tool_error_counts | Tool histograms (≥30% error rate on one tool = red flag) |
Cross-reference with the trajectory body's ## Result block, which
prints raw reward_info: db_match, db_reward, per-action
action_checks, and nl_assertions outputs. Those are the most direct
evidence for which component let you down.
Known Techniques That Improve tau2 Scores
Already wired in harness_config.yaml:
| Technique | Layer | Knob / file | Approx. lift |
|---|
| Phase-aware tool gating — restrict action tools in first N turns | Configuration | PhaseAwareToolFilter.read_only_steps (default 2); bump to 3 for policy-heavy domains | moderate (closes failure mode A) |
| Loop-detection — catch repeated identical tool calls | Configuration | LoopDetectionProcessor; tune threshold | moderate on confirmation-heavy tasks |
Hardened baseline toolkit — ParseRetryProcessor, ToolCallCorrectionLayer, ToolFailureGuard, TokenBudgetProcessor | Configuration | all in harness_config.yaml | small each, meaningful in aggregate |
| SystemAppendProcessor — append domain guidance without replacing tau2's policy prompt (see pattern below) | Instruction | recipe/tau2_evolver/system_append_processor.py + guidance_<domain>.md | moderate-large on policy-sensitive tasks |
PolicyHintProcessor — condition-triggered positive directives for telecom mobile_data_issue | Instruction | benchmarks/tau2/policy_hint.py (off by default) | large on the mobile-data subset |
| Extended Thinking + budget — Anthropic reasoning on Sonnet/Opus | Configuration | --extended-thinking --thinking-budget-tokens 62976 | substantial; a Sierra-highlighted axis |
Public-writeup techniques worth authoring as new candidates:
| Technique | Layer | What to write | Approx. lift |
|---|
| Policy rewrite → decision tree — recast domain policy into numbered Check→If→Then branches with explicit tool names and args | Instruction (guidance_<domain>.md) | Replace verbatim policy text with decision-tree form via SystemAppendProcessor | +15-22 pp documented — Quesma 2025 on GPT-5-mini; Barres replicated ~+20 pp on GPT-4.1 |
| Input-reformulation pre-pass (IRMA) — before each agent turn, rewrite the latest user message with the relevant policy snippet + shortlisted tools | Control (on_before_model processor) | Ship as output_dir/processors/irma.py | +12-19 pp on pass^5 (arXiv 2508.20931) |
| Plan-confirm-execute — planner emits JSON plan, asks one disambiguating question, executor issues the write | Control + Instruction | Plan-schema guidance rule + processor that blocks writes until plan is in state.slots | designed for irreversible writes; catches hallucinated IDs and basic-economy-modify attempts pre-execution |
| Trust-score message revision — score each outgoing assistant message and rewrite low-trust ones | Control (on_after_model processor) | Call cheap scorer or main model at low temp; rewrite below threshold | Cleanlab 2025: all-domain lift on GPT-5 |
| Dependency-aware reordering — when policy requires a different execution order than the user stated | Instruction | Guidance rule: "if X requires precondition Y, do Y first regardless of user order" | large on multi-request retail tasks |
Case-specific control processor — e.g. on_before_tool refusing writes when order-status vs verb contradict | Control | output_dir/processors/<name>.py | case-dependent (targets D or E) |
SystemAppendProcessor pattern
tau2 injects its own system prompt (the full policy document + tool
descriptions). Never replace it with TemplateSystemPromptBuilder —
that deletes the policy text and scores collapse. SystemAppendProcessor
reads tau2's system message on on_task_start, appends a markdown
guidance file, and sets task_system_prompt to the combined text.
processors:
- _target_: recipe.tau2_evolver.system_append_processor.SystemAppendProcessor
append_path: /abs/path/to/recipe/tau2_evolver/guidance_retail.md
The shipped guidance_retail.md encodes execute-on-stated-intent,
exchange-vs-modify disambiguation, and attempt-tool-calls discipline.
For airline/telecom, author a new file drawn from patterns seen in
that domain's trajectories — this is where the Quesma/Barres
decision-tree rewrite lives.
Common Failure Modes
A. Premature action call
Agent fires a write on turn 0 or 1 from the user's opening message
alone, before any read tool.
- Signal: first entry of
tool_call_counts is an action tool;
small num_messages; db_match=False.
- Fix: raise
PhaseAwareToolFilter.read_only_steps; or guidance
rule "always call at least one read before any action".
B. Over-confirmation / re-asking loop
Agent proposes a plan, user confirms, agent asks for re-confirmation,
exchange loops until max_steps.
- Signal:
num_messages ≥ 15 on a 4-6 turn task; many reads, no
writes; exit_reason = max_steps or user_stop.
- Fix: instruction rule "execute on stated intent, do not ask for
re-confirmation" (already in
guidance_retail.md); tighter
LoopDetectionProcessor.
C. Hallucinated arguments or unsupported claims
Two shapes:
- In a tool call: invented order ID / customer ID / parameter not
present in dialogue or read-tool output. Signal:
tool_error_counts
spikes on a mutation tool; body contains Order X not found.
- In an assistant message (no tool call): confident numeric or
policy claim the agent didn't look up. Signal:
nl_reward < 1.0
with db_match=True; no corresponding read tool in the turn.
Fix: guidance rule "every ID, number, or date must appear verbatim in
a prior tool result or user message"; or a trust-score / before-tool
processor that pattern-matches arguments against recent read outputs.
D. Exchange-vs-modify tool confusion (retail)
modify_pending_order_items on a delivered order, or
exchange_delivered_order_items on a pending order.
- Signal:
failed_actions names one of those tools; dialogue has
"change / update / exchange / swap / replace" without disambiguating.
- Fix: verb → tool + order status mapping (already in
guidance_retail.md); or a control processor refusing the wrong tool
when status contradicts.
E. Policy violation
Agent does something forbidden — swaps payment methods on a return,
refunds without auth, modifies basic-economy airline reservation,
issues a courtesy credit without escalation.
- Signal:
action_checks has an extra action or a missing
required auth step; db_reward < 1.0 even with db_match=True.
- Fix: surface the specific rule in
guidance_<domain>.md as an
explicit shortcut ("when X and policy Y, do Z, do NOT W"). The
decision-tree rewrite is the scalable version.
F. Stop-guard / user-sim boundary collisions
User sim sends "Yes, proceed.###STOP###" on the confirming turn; the
orchestrator terminates before the agent executes the write.
StopGuardUserSimulator (benchmarks/tau2/stop_guard.py) strips this
when the stripped content starts with yes\b — zero false positives on
a 114-task retail run, ~+8 pp on retail Sonnet-4.5.
- Signal:
exit_reason = user_stop, small num_messages, no
write in tool_call_counts, DB unchanged.
- Fix: pipeline issue, not agent bug. If a new stop-token
variant slips through, log to
_meta_scratch/NEEDS_FROM_HUMAN.md;
do not build an agent-layer fix for this.
G. Dependency-graph literalism
Agent follows the user's stated sequence literally when policy requires
a different order — e.g. closing an account before redirecting its
balance, when "no CLOSED accounts" is a precondition.
- Signal:
failed_actions names a write that's individually
correct but arrived in the wrong order; trajectory shows exact
user-dictation order; db_match=False on a multi-step request.
- Fix: guidance rule "derive the dependency graph from the policy,
execute preconditions first regardless of stated order"; or a
plan-confirm-execute structure that exposes the full plan first.
Red Flags — Pipeline Bugs, Not Agent Bugs
Record in _meta_scratch/NEEDS_FROM_HUMAN.md; do NOT build an
agent-layer candidate:
reward=0.0 with all action_checks.action_match=True and db_match=True → NL-assertion judge failed or timed out.exit_reason=error with no traceback → simulation harness swallowed an exception.tool_error_counts dominated by one tool across most tasks → that tool is broken upstream, not miscalled.cost_usd=0.0 on a non-zero-message run → cost-accounting hook lost data; run valid but budget signals unusable.- All three domains drop on the same commit → upstream tau2 regression (
~/tau2-bench git log); uniform rises may be the upstream audit, not your intervention.
- An airline task with a correct-looking trajectory scoring 0.0 → single-solution brittleness (τ² §5). One ≠ systemic; >3 in a round = escalate.
References
Pointers
benchmarks/tau2/README.md — architecture, env vars, install, HarnessX scores.benchmarks/tau2/tool_filter.py — PhaseAwareToolFilter.benchmarks/tau2/stop_guard.py — premature-STOP regex + false-positive analysis.benchmarks/tau2/policy_hint.py — telecom condition detectors + affirmative-hint templates.benchmarks/tau2/harness_config.yaml — hardened default pipeline.recipe/tau2_evolver/system_append_processor.py — instruction-layer injection.recipe/tau2_evolver/guidance_retail.md — reference domain-guidance file.recipe/tau2_evolver/defaults.py — NUM_TRIALS, split, model tiers, evolve-loop constants.
