| name | research-preference |
| description | Compare and rank candidate ML research ideas, experiment plans, or code branches before expensive evaluation. Use when the user asks which experiment to run next, which branch deserves a limited GPU or time budget, how prior runs change the choice, or what low-cost pilot would resolve a close decision. Best for two or more frozen candidates sharing a measurable objective; direct testing, ideation, literature review, and result summarization remain separate workflows. |
| license | MIT |
| metadata | {"author":"leo-lilinxiao","version":"0.2.1"} |
Research Preference
Allocate the next unit of research compute to the candidate with the strongest evidence-adjusted branch value. Make a relative, traceable choice before full execution.
The instruction-only path works in Agent Skills-compatible clients. The bundled validation and aggregation CLI uses Python 3.10 or newer and the standard library.
Workflow
1. Frame the allocation
Extract these fields from the request and available artifacts:
- two or more candidate IDs;
- shared objective, metric direction, and hard constraints;
- scarce budget and the cost of one full evaluation;
- prior attempts visible at decision time;
- the action requested: analyze, design a pilot, run a pilot, or select for full evaluation.
Use a cheap definitive measurement immediately when it fits inside the budget. Keep multi-objective choices provisional until the user supplies the tradeoff or weights.
2. Freeze candidates
Snapshot every candidate before judging:
- assign neutral, stable IDs;
- preserve the submitted plan, code, config, or content hash;
- represent each candidate with the same fields and comparable detail;
- separate candidate generation from the selection pass.
For three or more candidates, persistent decisions, or repeatable evaluation, create a decision packet using assets/decision-packet.schema.json. Start from assets/example-decision.json when useful.
Resolve this skill's root from the location of this SKILL.md, then validate the packet:
python3 <skill-root>/scripts/research_preference.py validate decision.json
Treat validation errors as input work to resolve before comparison.
3. Build the evidence ledger
Classify every decision-relevant claim as one of:
measurement — measured under a declared protocol;artifact — verified code, config, log, or output;history — outcome from a related earlier branch;static-analysis — feasibility or correctness evidence;assumption — an explicit prior awaiting measurement.
Check metric definition, data split, seed policy, preprocessing, compute budget, and evaluation stage. Mark each item comparable: true only when it can support a direct candidate contrast. Record candidate, harness, dependency, data, and infrastructure failures by their actual source.
4. Choose the evidence mode
- Preference mode — default: compare plans, code, artifacts, and prior results. Use it when the evidence already supports a useful choice.
- Pilot mode — escalation: gather one low-cost observation when a specific uncertainty could reverse the preference. Read
references/pilot-experiments.md before designing or running it.
An analysis request produces a pilot proposal. An execution request supports an authorized, in-scope pilot run.
5. Judge pairwise
Establish the eligible pool from the hard constraints. Keep fundamental task mismatch distinct from local repair effort.
Compare each decisive pair on:
- task and metric fit;
- quality and comparability of prior evidence;
- correctness and feasibility;
- realistic upside over the next few iterations;
- repair effort and clarity of the upgrade path;
- novelty relative to earlier branches;
- full-run cost, failure risk, and opportunity cost.
State the preference as A, B, or tie, followed by a concise evidence-grounded rationale and the observation that would reverse it. Use categorical confidence: high, medium, or low.
For a two-candidate decision, repeat the decisive comparison with labels and presentation order reversed. For larger pools or close choices, read references/preference-judging.md, write one JSON object per judgment following assets/comparison.schema.json, and aggregate the judgments:
python3 <skill-root>/scripts/research_preference.py validate \
decision.json --comparisons comparisons.jsonl
python3 <skill-root>/scripts/research_preference.py aggregate \
decision.json comparisons.jsonl --require-complete \
--output preference-summary.json
The aggregator exposes pairwise coverage, Copeland ranking, a Condorcet winner, presentation-order conflicts, and preference cycles. Use these diagnostics as decision evidence; ground the final choice in the underlying rationale and sources.
Interpret CLI exit codes as 0 for success, 2 for input or usage repair, and 3 for missing pairwise coverage under --require-complete. Read references/cli.md for the complete command, output-safety, and presentation-order contract.
6. Decide the next spend
Choose one outcome:
selected — evidence supports the next full evaluation;provisional — one candidate leads while a reversal condition remains open;tie — candidates remain indistinguishable under current evidence;direct-evaluation — the definitive test is already the efficient next step;blocked — the decision frame or evidence pipeline needs repair.
Use references/decision-record.md when the user requests a durable Markdown or JSON record. Read references/foundations.md when explaining the method or adapting it to a new domain.
Response contract
Return this compact structure:
## Decision
<status and selected candidate>
## Why
<decisive pairwise evidence and comparability>
## Candidate state
<eligible candidates, fundamental issues, and repairable work>
## Reversal condition
<single observation most likely to change the preference>
## Next spend
<full evaluation, one pilot, evidence repair, or stop>
## Confidence
<high | medium | low, tied to evidence quality>
Completion check
- Candidate snapshots remained fixed throughout selection.
- Objective, metric direction, constraints, and budget are explicit.
- Evidence links resolve to visible artifacts or clearly labeled assumptions.
- Comparability and failure provenance are explicit.
- Decisive comparisons remain stable after order reversal, or confidence reflects the sensitivity.
- The next action stays within the user's requested scope and budget.
- The conclusion is scoped to the declared objective, evidence, and decision boundary.
