| name | legalsearchqa-eval |
| title | legalsearchqa-eval |
| description | Evaluates a system's ability to retrieve up-to-date legal information from external sources and reason over it to answer multiple-choice legal questions. It probes factual accuracy, uncertainty calibration, and evidence grounding in dynamic legal domains like federal executive orders and tax provisions. Use when the user wants to benchmark on LegalSearchQA, or asks about evaluating this task. Reports Accuracy. |
| author | qhjqhj00 |
| author_url | https://github.com/qhjqhj00/research-skills-pool/tree/master/skill-factory/output/legalsearchqa-eval |
| license | MIT |
| version | 0.1.0 |
| execution_mode | open |
| jurisdiction | us |
| practice | general |
| language | en |
legalsearchqa-eval
L-MARS: Legal Multi-Agent Workflow with Orchestrated Reasoning and Agentic Search — Wang et al. (2025) (arXiv:2509.00761, 2025)
What this evaluates
Evaluates a system's ability to retrieve up-to-date legal information from external sources and reason over it to answer multiple-choice legal questions. It probes factual accuracy, uncertainty calibration, and evidence grounding in dynamic legal domains like federal executive orders and tax provisions.
Datasets
Metrics
Accuracy (primary) — range: [0, 1]
- Fraction of multiple-choice questions answered correctly against expert-annotated ground truth.
U-Score — range: [0, 1]
- Rule-based uncertainty metric: 0.25H + 0.20T + 0.25*(1-C) + 0.15*(1-J) + 0.15*(1-D), where H=hedging, T=temporal vagueness, C=citation sufficiency, J=jurisdictional specificity, D=decisiveness. Ranges [0,1]; lower is better.
LLM-as-Judge — range: categorical (low/moderate/high)
- GPT-o3 evaluates answers on factual accuracy, evidence grounding, clarity of reasoning, and uncertainty calibration. Each response receives a holistic rating of low, moderate, or high, determined by majority vote.
Input / output format
Input: A legal multiple-choice question referencing 2025 legal status, requiring retrieval from external sources.
Output: A multiple-choice answer and a supporting explanation, aggregated into structured JSON.
Scoring recipe
def compute_metrics(predictions, gold_answers):
accuracy = sum(1 for p, g in zip(predictions, gold_answers) if p == g) / len(predictions)
H = measure_hedging(predictions)
T = measure_temporal_vagueness(predictions)
C = measure_citation_sufficiency(predictions)
J = measure_jurisdictional_specificity(predictions)
D = measure_decisiveness(predictions)
u_score = 0.25*H + *T + *(-C) + *(-J) + *(-D)
judge_ratings = [gpt_o3_judge(q, p) q, p (questions, predictions)]
final_rating = mode(judge_ratings)
accuracy, u_score, final_rating
