preference-calibrate

name

preference-calibrate

description

Elicits scenario-bounded human preferences through controlled alternatives, records literal choices and provenance, and turns supported signals into a reusable preference brief. Use when a decision-maker can reject work but cannot yet specify what to change, when repeated subjective directions have failed, or before delegating visual or editorial choices. Do not use to score taste or replace correctness, accessibility, task fitness, or human judgment.

Preference Calibrate

Turn a vague rejection into a narrow, testable preference record. Do not choose for the decision-maker.

Keep three judgments separate

Keep correctness, task fitness, and human preference separate. Never average them into one quality or taste score. A preference may guide choices only inside its recorded scenario and may never override an objective blocker.

Route before calibrating

1. Name the artifact type, scenario, axis, audience, task, decision-maker role, and exact revision.

2. Run the bundled router when the CLI is available:

   node scripts/preference-harness.mjs route \
     --scenario marketing-web \
     --axis composition \
     --json
   

3. Stop on NO_EXACT_ROUTE or AMBIGUOUS_ROUTE. Never fall back to a generic design checklist.

4. Load references/calibration-protocol.md for the full evidence and comparison contract.

Calibrate one unresolved axis

1. Write the exact decision the comparison must support.
2. List correctness, accessibility, legal, brand, and task constraints first.
3. Select the smallest unresolved preference axis.
4. Create two to six alternatives with neutral alt-NN identifiers.
5. Keep content, data, constraints, viewport, and environment fixed.
6. Declare the factor values for every alternative. The changed factor key must match the routed axis exactly.
7. Mark the trial confounded when more than one factor changed. Name every known confound.
8. Randomize externally when appropriate and always record the presentation order without leading labels. The dossier proves the recorded order, not the randomization method.
9. Allow choose, reject_all, and abstain.
10. Record the literal response before writing any interpretation.
11. Preserve corrections by adding a response with supersedes; never overwrite the earlier response.

Extract a bounded rule

Create a rule only from an active calibration response that is not an abstention.

Every rule must include:

• the exact statement;
• whether it uses human wording or agent inference;
• scenario and axis;
• conditions where it applies;
• explicit exclusions;
• source response IDs;

Version 0.1 records exactly one active rule per dossier. A corrected rule is a new fragment whose supersedes points to the active rule. Unrelated rules use separate dossiers. rule add creates a separate append-only rule_states entry with state tentative; never put mutable state inside the rule fragment. After held-out assessment, append a superseding state of supported, contradicted, or insufficient_signal and cite that preference-assessment ID.

Keep a rule tentative after a confounded comparison or historical self-report. Never silently merge contradictory rules from different scenarios.

Test transfer

1. Prepare a held-out artifact that did not appear during calibration.
2. Commit the predicted outcome before showing the artifact.
3. Test exactly one rule per held-out trial in version 0.1.
4. Reject reused evidence hashes.
5. Record the new response literally.
6. Assess correctness, task fitness, and preference independently.
7. Report a raw match, mismatch, or inconclusive result. One match is one observation, not a percentage or general law.

Stop and refuse

Return INSUFFICIENT SIGNAL and the next smallest comparison when:

• no human verdict or explicitly synthetic oracle exists;
• candidates cannot be tied to exact revisions;
• objective requirements fail;
• the comparison is biased or cannot be separated;
• the decision-maker rejects all candidates or abstains;
• held-out evidence is unavailable when transfer is claimed.

Refuse to:

• select what looks best on the decision-maker's behalf;
• infer preferences from trends, popularity, or model votes;
• emit a universal taste score;
• transfer a marketing preference to a product workflow;
• publish a private path, unsanitized response, or unapproved quote.

Output

Produce:

1. A machine-readable dossier validated by preference-harness validate.
2. A concise brief of scoped rules, exclusions, conflicts, unresolved questions, and held-out results.
3. A report that labels artifact origin, verdict origin, and intended claim use.

Use references/dossier-contract.md when constructing fragments or reviewing a dossier manually.