zicato-build-board

name

zicato-build-board

description

The board-builder copilot's deep board-craft guide — designing board entries, judges, and the weighted loss through a GUI as a DRAFT applied on confirmation. Use when a copilot is helping an operator design a board, declare judges, shape the loss to express an objective, hold out a slice, or fix a toothless eval. Covers entries (single/multi-turn, expectations, tags incl. holdout, weight), judges (declared/in-run, judge_name, per_judge_weights, board_meta, the collusion-guarded emulator), and how the loss knobs combine to express an objective. Defers the exact scalar formula to SCORING.md, the schema to BOARD-FORMAT.md, anti-overfitting to OVERFITTING.md, discrimination diagnosis to LOOP-HEALTH.md.

Building a zicato board (copilot guide)

This skill is for the board-builder copilot — the agent helping an operator design a board through the builder GUI: its entries, its judges, and the weighted loss that turns runs into the tournament scalar. It is the deep board-craft skill, grounded in the design docs but framed as the builder workflow. It is the board sibling of zicato-build-tournament (the whole-contract walkthrough — structure, holdout split, proposer, gate). Defer:

• The on-disk JSONL schema (every field, validation, RunResult) → BOARD-FORMAT.md.
• The exact scalar formula and the gate → SCORING.md.
• The train/holdout split, the Ladder budget, leakage → OVERFITTING.md.
• Diagnosing a board that can't discriminate → LOOP-HEALTH.md and zicato-diagnose-health.

The design-principles siblings remain the authority on what belongs on a board (zicato-design-boards) and what to measure (zicato-design-judges); this skill is how the copilot operates the builder over those principles. See ../AGENTS.md for the operating rules.

Board entries — the unit the copilot edits

Each board entry is one task the inner harness runs. Three live kinds; pick the cheapest that carries the behavior's signal (full table and cost/noise trade in zicato-design-boards):

• single_turn — one input message → the agent's final reply. Cheapest, fully deterministic input. The default; reach for it first.
• multi_turn_scripted — a fixed list of user turns replayed verbatim. Cheap and deterministic; use it to regression-test a known revision sequence.
• multi_turn_emulated — an aux-LLM user_persona plays the user, reacting to the agent. Expensive and stochastic; reserve it for behaviors that only emerge when the user pushes back.

Every entry carries an envelope the copilot's tools edit:

• input / turns / user_persona — the per-kind task content.
• expectation — the entry's single typed OUTCOME check, evaluated post-hoc on the run's product. Two families compile to it: Predicate (deterministic — predicate dotted path, expected_text, regex, json_schema) and Rubric (LLM-graded, runs on the aux callable, carries threshold/scale; threshold: null = advisory). An entry has at most one expectation; absent → the entry is scored on drift loss alone. reads selects the slice graded (final_output vs conversation_end) — and must point at the agent's real output, never a self-summary / proxy field. Prefer a deterministic matcher whenever the behavior is code-checkable — it adds zero noise to pass-rate. The predicate seam also accepts a scorer: a callable returning a continuous float in [0,1] (or a (float, metrics) tuple like {"precision": .., "recall": ..}) gives the entry a graded per-entry score instead of a bare pass/fail — reach for it when partial credit discriminates (a 0.6-vs-0.4 delta two candidates a bool would tie). A bool matcher leaves the entry binary (score=None → the reducer derives 1.0/0.0); the score + metrics ride out to loss.json and feed the failure-mode channel below.
• tags — operator labels pattern detectors and the brief slice on. The holdout tag is load-bearing: it marks the entries withheld from the promotion signal and confirmed afterward (the train/holdout split — see "Hold out a slice" below).
• weight — the entry's pull on the aggregate (default 1.0).

Defer the exact field shapes and validation to BOARD-FORMAT.md.

Judges — the in-run PROCESS axis

Where an expectation asks "is the product right?", a judge asks "is the process clean?" — it watches the live reasoning stream and, on an adverse verdict, emits a goldfive custom drift tagged with the judge's name. The two axes are independent and combine into the scalar under separate coefficients (pass-rate vs drift-loss), so a behavior you care about on both gets both an expectation and a judge. Decide the axis with zicato-design-judges.

What the copilot sets on each judge:

• name — a stable, board-unique slug. It becomes goldfive's judge_name, and per_judge_weights keys on it — choose it deliberately, because the loss weighting reaches the judge by this name.
• mode — inline (a natural-language criterion graded by the aux LLM) or python (a dotted path to a deterministic process-judge callable).
• body — the criterion text (inline) or the dotted path (python).
• severity — info / warning / critical; scales how hard a violation weighs (default 1 / 3 / 10 via severity_weights).

Grade the tool ledger, not the narration. A python judge that scores what the agent did reads goldfive's real tool-call ledger — its ctx.session_state.recent_events (kind == "tool_observed": tool_name, result_preview, error_message) — NOT the model's reasoning text. Custom judges fire only at reasoning observation points, so a judge that scans the transcript for tool names grades the story the agent told (and fires on chatter that merely names a tool), not the round-trips it ran. The copilot steers an operator authoring a deliverable-grading judge to the ledger source; worked example file_findability in examples/zicato_examples/target_1_presentation/judges.py.

Two more board-wide judge facts the copilot surfaces via the board_meta header (set when non-default):

• disable_drift — suppress goldfive's built-in detectors by kind, board- wide (e.g. you script user_steer turns, so steer-drift is benign noise). Custom judges are removed by deleting them from judges, never here.
• judge_only — keep goldfive judging but turn off all steering. Use it to measure the bare, un-steered harness, or when evolving the steerer-free path; your judges and expectations then carry the entire signal.

The collusion guard (design constraint the copilot must respect): every grader — rubric matcher, inline/aux judge, the user-emulator — runs on the auxiliary callable, distinct from the harness callable that runs the system under test (zicato hard-errors on an identity match). The emulator sees only the persona + the user-facing transcript, never the entry's expectation. The implication for board craft: write personas to withhold the answer (so the entry tests whether the harness can reach the goal, not read it off the persona), and never author a judge that reaches into harness state — judge only the observable stream.

The weighted loss — shape it to express an objective

The tournament scalar is a weighted combination of a drift-loss term and a pass-rate term. The exact formula, the three-rule gate, and the defaults live in SCORING.md — do not restate it. What the copilot teaches is how to shape the loss with the knobs to express what the operator means by "better":

Knob	Keys on	Use it to express…
drift_weight / pass_weight	the two top-level axes	whether cleanliness or correctness dominates (pass_weight = 0 ignores expectations; drift_weight = 0 scores on pass-rate alone)
severity_weights	info / warning / critical	how much worse a critical is than an info
per_kind_weights	built-in DriftKind token	"this harness's off_topic matters 2× the others"
per_judge_weights	judge name	telling custom judges apart — they all share the custom kind, so this is the only per-judge lever
default_judge_weight	—	the fallback for a custom judge absent from per_judge_weights
plan_revision_weight	—	how hard mid-run plan churn (the agent rewriting its own plan) weighs in the scalar
runtime_weight	—	opt-in pressure toward faster runs (0 keeps wall-clock out of the loss)
namespace_weights	namespace prefix (drift:, cost:, rubric:, …)	the multi-objective scalar — signed coefficients fold each namespace into the scalar
pass-rate monotonicity (pass_rate_monotonicity bool + pass_rate_monotonicity_scope)	the gate	"a challenger may never break an entry the champion passed" — per_entry scope (default; no individual passing entry may regress — invariant/regression boards) vs aggregate (only the board-wide pass-rate may not drop — sampled evaluation boards). The bool is the on/off switch; there is no off scope

Worked example of shaping: to say "I care most that the agent stays on topic, and I will not promote anything that breaks a passing entry," the copilot sets per_kind_weights["off_topic"] = 2.0 (elevate that drift kind), leaves the other kinds at the uniform default, and leaves pass-rate monotonicity on (the gate then hard-rejects any pass-rate regression). To weight a single custom judge — say cite_before_metric is a real defect but ack_before_edit is a nicety — set per_judge_weights = {"cite_before_metric": 2.0, "ack_before_edit": 0.5}. The copilot's job is to translate the operator's objective into the minimal set of knob changes and show the resulting loss shape; the numeric calibration loop (run an epoch, check the journal, retune) is in zicato-tune-scoring.

Designing a discriminating board

A board only earns its cost if its entries separate champion from challenger. The copilot designs for discrimination, not just coverage:

• Avoid the toothless eval. An entry that returns the same drift loss and the same pass/fail for every generation is dead weight — it can never break a tie. The failure modes are all-pass (expectations too lenient), all- fail (task beyond the harness, or predicate over-specified), and constant drift (often telemetry not reaching the reducer). A good entry sits in the sensitive band: the champion is on the edge, so a real improvement shows up as a score delta. The copilot should cross-reference LOOP-HEALTH.md / zicato-diagnose-health and steer the operator to design a board that passes zicato health before trusting a verdict.
• Span behaviors, not inputs. Fifty near-identical prompts carry one behavior's signal at fifty times the cost; enumerate the distinct behaviors to defend and put one or two discriminating entries on each.

Hold out a slice — don't let the proposer memorise the board

Because the board is the whole signal the proposer optimises against, a challenger can win by memorising the board rather than generalising — its score on the entries it has seen improves while its true quality does not. The defenses the copilot builds in:

• Hold out a slice. Tag ~30% of entries holdout; they are withheld from the round's promotion signal and the gate confirms the winner on them afterward. A challenger that won on train but does not hold up on the holdout has a generalization gap — it over-fit. Set the split with set_holdout.
• Don't over-specify. An entry pinned to an exact string or a brittle regex is easy to satisfy in a way that doesn't generalise; prefer a predicate or a rubric that captures the behavior, not the surface form.

The full anti-overfitting design — why the holdout is query-budgeted (Ladder / Thresholdout), what leakage restriction means, how the generalization gap is read — is in OVERFITTING.md. The builder's job is to make the split easy to set and to remind the operator it is the insurance against a board the proposer can game.

The same train-slice the split defines also feeds an opt-in proposer failure-mode feedback channel: a board-anonymized, bucketed summary of outcome marginals (board-wide rates like "over-retrieves ~40% of runs", plus precision/recall from a scorer's metrics) computed over the train slice only — never the holdout, never any entry id / question / output token. It feeds the proposer the marginal, never the joint, so the proposer can target why answers are wrong without being able to reconstruct the board. The operator's scoring.json outcome_summarizer_spec hook can contribute extra marginals (numeric-only, sanitized + bucketed before they reach the proposer). The channel reuses the existing holdout split + bucketing; see zicato-design-experiment.

The copilot's operating contract — DRAFT, then apply

The board builder edits a draft board, never the live epoch's board directly. The copilot's tools (conceptual):

Tool	Edits the draft…
edit_board_entry	add / edit / remove one entry (input, expectation, tags, weight)
add_judge	declare a process judge on an entry (name, mode, body, severity)
set_weights	the loss knobs above — the scalars (drift_weight, pass_weight, default_judge_weight, plan_revision_weight, runtime_weight) and the wholesale maps (severity_weights, per_kind_weights, per_judge_weights)
set_namespace_weights	the multi-objective namespace coefficients (namespace_weights) + the diff_complexity_weight parsimony term
set_holdout	the train/holdout split (holdout-tagged ids and/or fraction) + the wider anti-overfitting block (Ladder, rotation, placebo cadence — detail in zicato-build-tournament)
validate	(read-only) re-runs board validation (ids unique + filesystem-safe, per-kind fields present, expectation reads valid for the kind, judge slugs unique, disable_drift tokens resolve) — plus the statistical margin-vs-noise-floor rule when the epoch carries a measured A/A floor
preflight	(read-only) measures whether THIS board can out-signal its own noise — the A/A floor vs a deliberate-degradation signal; a warn verdict means the board is saturated (cannot discriminate even a broken tree). CLI: zicato board preflight / board audit

The loop on every operator request:

1. Edit the draft with the matching tool.
2. Preview — show the changed entry / judge / weight and any validate warning (a leniency smell, a duplicate id, an unanchored regex).
3. State the consequence — the board is part of the frozen contract, so applying it rolls the epoch (generations answering a different board are not comparable). Batch board changes so the epoch rolls once.
4. Apply only on explicit confirmation, and never apply a draft that fails validate.

The board copilot, like the tournament copilot, never starts a live zicato evolve — it produces a validated board, and the live run is the operator's separate explicit go-ahead (see ../AGENTS.md).

The direct GUI — the board editor

Everything above is also reachable WITHOUT the copilot, through the dashboard's board editor (the Board section of the builder view). Each entry is a row that opens an inline accordion form covering the full schema for its kind: the common fields (id, kind, budget, weight, tags), the per-kind discriminant (single-turn input; scripted turns list; emulated persona; adversarial spec + required drift kinds), an optional expectation sub-form (per matcher — text/ regex/JSON-schema textarea, dotted-path predicate, or a structured rubric that serializes to the JSON spec), and a judges list editor. A board-meta panel sets the board-level disable_drift + judge_only header, and a paste-JSONL import box bulk-loads entries (a leading board_meta header line routes to the meta panel).

The editor drives the SAME ops the copilot's tools do — Save posts the whole entry through edit_board_entry, Delete through remove_board_entry, a judge badge's × through remove_judge, the meta panel through set_board_meta. The weighted loss has its own controls in the Weights section: the scalar rows plus wholesale mapping editors for severity_weights / per_kind_weights / per_judge_weights (the per-judge rows are SEEDED from the judges declared on the board, with a free-text add-key row for a judge not yet on it) and the signed namespace_weights with an add-key row — each posting the WHOLE mapping through set_weights / set_namespace_weights. It carries no client-side validation twin: the server's field-precise ValueError renders verbatim in an inline strip, and the editor stays open to fix it. The id is locked while editing (Duplicate seeds a fresh id), and the holdout tag is owned by the train/holdout toggle, never the tags input. When you help an operator over the GUI, name the CONTROL and the CONSEQUENCE (the board is contract — a save rolls the epoch), the same discipline as the copilot loop above.

A good board build

• Design for discrimination first. A board that can't tell candidates apart is worse than a small one that can — cut dead-weight entries, target the sensitive band, run zicato health.
• Put the behavior on the right axis. Product correctness → an expectation; trajectory quality → a judge. Both when you care about both.
• Shape the loss to the objective with the minimal knob change, then hand the numeric calibration to zicato-tune-scoring.
• Hold out ~30% and prefer behavior-capturing matchers over brittle exact strings, so the proposer can't memorise its way to a promotion.
• Batch edits and apply once — every board change rolls the epoch.