legalsearchqa-eval

name	legalsearchqa-eval
title	legalsearchqa-eval
description	Evaluates a system's ability to retrieve up-to-date legal information from external sources and reason over it to answer multiple-choice legal questions. It probes factual accuracy, uncertainty calibration, and evidence grounding in dynamic legal domains like federal executive orders and tax provisions. Use when the user wants to benchmark on LegalSearchQA, or asks about evaluating this task. Reports Accuracy.
author	qhjqhj00
author_url	https://github.com/qhjqhj00/research-skills-pool/tree/master/skill-factory/output/legalsearchqa-eval
license	MIT
version	0.1.0
execution_mode	open
jurisdiction	us
practice	general
language	en

legalsearchqa-eval

L-MARS: Legal Multi-Agent Workflow with Orchestrated Reasoning and Agentic Search — Wang et al. (2025) (arXiv:2509.00761, 2025)

What this evaluates

Evaluates a system's ability to retrieve up-to-date legal information from external sources and reason over it to answer multiple-choice legal questions. It probes factual accuracy, uncertainty calibration, and evidence grounding in dynamic legal domains like federal executive orders and tax provisions.

Datasets

• LegalSearchQA — total 200; splits: test (200); repo https://github.com/boqiny/L-MARS

Metrics

• Accuracy (primary) — range: [0, 1]
  • Fraction of multiple-choice questions answered correctly against expert-annotated ground truth.
• U-Score — range: [0, 1]
  • Rule-based uncertainty metric: 0.25H + 0.20T + 0.25*(1-C) + 0.15*(1-J) + 0.15*(1-D), where H=hedging, T=temporal vagueness, C=citation sufficiency, J=jurisdictional specificity, D=decisiveness. Ranges [0,1]; lower is better.
• LLM-as-Judge — range: categorical (low/moderate/high)
  • GPT-o3 evaluates answers on factual accuracy, evidence grounding, clarity of reasoning, and uncertainty calibration. Each response receives a holistic rating of low, moderate, or high, determined by majority vote.

Input / output format

Input: A legal multiple-choice question referencing 2025 legal status, requiring retrieval from external sources.

Output: A multiple-choice answer and a supporting explanation, aggregated into structured JSON.

Scoring recipe

def compute_metrics(predictions, gold_answers):
    accuracy = sum(1 for p, g in zip(predictions, gold_answers) if p == g) / len(predictions)
    H = measure_hedging(predictions)
    T = measure_temporal_vagueness(predictions)
    C = measure_citation_sufficiency(predictions)
    J = measure_jurisdictional_specificity(predictions)
    D = measure_decisiveness(predictions)
    u_score = 0.25*H + *T + *(-C) + *(-J) + *(-D)
    judge_ratings = [gpt_o3_judge(q, p)  q, p  (questions, predictions)]
    final_rating = mode(judge_ratings)
     accuracy, u_score, final_rating