| name | answer-matching-evaluation |
| title | Answer Matching Outperforms Multiple Choice for Language Model Evaluation |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2507.02856 |
| keywords | ["Evaluation Methods","Benchmark Flaw Detection","Free-Form Responses","LLM Judging","Shortcut Learning"] |
| description | Evaluate language models using open-ended answer generation and semantic matching instead of multiple choice, eliminating test-taking shortcuts and achieving near-perfect alignment with human judgment. |
Answer Matching: Eliminating Shortcut Vulnerabilities in Language Model Evaluation
Multiple choice benchmarks like MMLU have become standard for evaluating large language models, but they contain a critical vulnerability: answers can often be selected without fully comprehending the question. This occurs because language models can learn statistical shortcuts—patterns in option formulations, letter distributions, or option lengths—that correlate with correct answers. Answer matching eliminates this vulnerability by asking models to generate free-form responses and comparing them semantically against reference answers, capturing genuine understanding rather than test-taking ability.
The shift from multiple choice to answer matching represents a fundamental change in evaluation methodology. Instead of constraining models to select from given options, the approach asks "what is your answer?" and then uses another language model to judge whether the generated response matches the reference. Human annotation studies show this method achieves near-perfect alignment with human grading, while multiple choice approaches often diverge significantly from human judgments.
Core Concept
Answer matching operates on three principles: generate freely, compare semantically, and judge holistically. Models produce unrestricted responses to questions without seeing answer options. These responses are then compared against reference answers using a language model as judge, which evaluates whether the generated answer is semantically equivalent to the reference, regardless of exact wording. This separates genuine understanding from pattern exploitation.
The evaluation framework is model-agnostic—any LLM can serve as the judge, making it scalable and adaptable to different settings. The judge must assess meaning equivalence, accounting for synonyms, rephrasing, and different but correct formulations. This requires semantic reasoning rather than exact matching, which naturally aligns with how humans grade open-ended questions.
Architecture Overview
The answer matching system comprises four components:
- Question Presenter: Provides questions to models without revealing answer options or hints
- Response Generator: Models being evaluated produce free-form, open-ended answers
- Judge LLM: Another language model tasked with comparing generated responses to reference answers semantically
- Scoring Aggregator: Compiles judge outputs into correctness scores and model rankings
The architecture is deliberately simple to maximize reproducibility and minimize judge bias. The judge's role is binary semantic equivalence determination, not complex reasoning or partial credit assignment.
Implementation
Begin by preparing evaluation questions and reference answers without multiple choice options:
