| name | sealqa-reasoning-search |
| title | SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models |
| version | 0.0.2 |
| engine | skillxiv-v0.0.2-claude-opus-4.6 |
| license | MIT |
| url | https://arxiv.org/abs/2506.01062 |
| keywords | ["benchmark","reasoning","search-augmented LLMs","information retrieval","noisy data"] |
| description | Evaluate search-augmented language models on fact-seeking questions with conflicting or unhelpful search results, revealing critical reasoning gaps in frontier models and testing robustness to noisy information. |
SealQA: Raising the Bar for Reasoning in Search-Augmented Language Models
Core Concept
SealQA is a benchmark designed to evaluate language models that augment their reasoning with web search capabilities. The benchmark specifically targets scenarios where web search yields conflicting, noisy, or misleading information, exposing the gap between models' advertised reasoning capabilities and their actual performance under realistic conditions.
The benchmark reveals that even frontier reasoning models struggle significantly (17.1% to 6.3% accuracy on challenging variants), demonstrating that naive search integration fails when faced with information clutter, contradictions, and irrelevant results—common in real-world information retrieval.
Architecture Overview
- Three Evaluation Variants: Seal-0 (baseline), Seal-Hard (aggressive noise), and LongSeal (extended context)
- Conflict Detection: Questions designed where search results contradict each other or the ground truth
- Long-Context Reasoning: Extended documents requiring deep multi-document understanding
- Needle-in-Haystack Scenarios: Evaluate models' ability to identify relevant information amid distraction
- Retrieval Robustness: Test resilience to noisy search results, ranked incorrectly, or tangentially related
- Reproducibility Focus: Public benchmark on Hugging Face for comparative evaluation
Implementation
The following steps outline how to construct and evaluate search-augmented reasoning systems:
- Define fact-seeking questions - Create question sets where web search alone is insufficient
- Generate diverse search results - Collect results with varying relevance, conflicts, and noise profiles
- Implement search integration - Add retrieval modules to the base language model
- Evaluate reasoning quality - Measure accuracy on questions requiring multi-document synthesis
- Measure robustness - Test performance degradation as search result quality decreases
- Analyze failure modes - Identify where models are misled by conflicting information
from typing import List, Dict, Any
import json
class :
():
.model_name = model_name
.search = search_engine
() -> [[, ]]:
results = .search.query(query, top_k=top_k * )
filtered = ._deduplicate_and_rank(results)[:top_k]
filtered
() -> []:
seen = ()
unique = []
r results:
content_hash = (r.get(, ))
content_hash seen:
seen.add(content_hash)
unique.append(r)
(unique, key= x: x.get(, ), reverse=)
() -> :
context = .join([ i, r (search_results)])
prompt =
reasoning = {
: question,
: search_results,
: ._generate_answer(prompt),
: ._estimate_confidence(search_results)
}
reasoning
() -> :
() -> :
results:
agreement_scores = [r.get(, ) r results]
(agreement_scores) / (agreement_scores)
() -> :
correct =
total = (benchmark_questions)
q_data benchmark_questions:
question = q_data[]
ground_truth = q_data[]
search_results = q_data[]
reasoning = .synthesize_reasoning(question, search_results)
predicted = reasoning[]
._match_answer(predicted, ground_truth):
correct +=
accuracy = correct / total total >
{: accuracy, : total, : correct}
() -> :
predicted.lower().strip() == ground_truth.lower().strip()
