| name | check |
| description | Analyzes or compares 2+ papers or methods. Triggers on "compare these papers", "analyze this method", "which is better?". |
check — analysis & comparison
Process
- Build a comparison matrix. For each paper/method under comparison, capture: method, dataset, metric, reported result, compute cost (if stated), and year.
- Check genuine comparability first. Before drawing any conclusion, verify the papers are actually comparable — same dataset/benchmark, same eval protocol, similar scale. Flag apples-to-oranges comparisons explicitly rather than silently normalizing them.
- Surface tradeoffs, not verdicts. State what each approach wins and loses on (e.g. accuracy vs. latency vs. training cost). A tradeoff table beats a single ranking.
- Never declare an unqualified "winner." If asked "which is better," answer relative to a named axis: "better on X, worse on Y" — never a bare superlative.
- Write output to
_papers/ as a comparison note containing the matrix and tradeoff summary, with a Markdown link citation to every paper compared.
Anti-Rationalization
- "Paper B has a higher benchmark number, so it's better" — a higher number on a different eval setup (different dataset split, different hardware, different task difficulty) is not a valid comparison; check the setup first.
- "The user wants a simple answer" — a false single-axis verdict is worse than a two-line tradeoff table; give the real answer.
- "They're close enough to compare directly" — state the comparability caveat explicitly instead of silently assuming it away.
Evidence
A comparison is complete when: the matrix has method/dataset/metric/cost/year for each entry, comparability caveats are explicitly stated where setups differ, and every paper in the matrix is cited with a Markdown link.
Red Flags
- A "winner" stated without naming the comparison axis
- Comparing numbers across different datasets/eval protocols without flagging it
- Missing compute cost or scale info when it materially affects the comparison
