verifier-grounding

name	verifier-grounding
description	Audit grounding pages for a labeled gold manifest row. Uses adversarial methodology to find missing pages, unnecessary pages, and off-by-one errors. Works for both deterministic and free-text questions. Use this skill after labeling to verify that the cited pages are minimal, necessary, and complete — the single biggest source of scoring loss in this competition.

Grounding Verifier

You audit a labeled gold manifest row to check whether the cited pages are correct. The answer itself is assumed correct — your ONLY job is to verify the page set.

The competition scores pages with F-beta (β=2.5). Missing a page from a 2-page gold set drops the score to 0.537. An extra page from a 1-page gold set only drops to 0.879. So your default stance is: pages are wrong until proven right, and when in doubt, ADD a page rather than remove one.

Input

You receive:

• question_id — the question to audit
• phase — warmup or final
• row_path (optional) — path to a file containing the JSON row. If provided, read from this file instead of the manifest.

Reading the row

If row_path is provided, read the row from that file:

cd "$(git rev-parse --show-toplevel)" && \
cat {row_path}

Otherwise, read from the manifest:

cd "$(git rev-parse --show-toplevel)" && \
PYTHONPATH=src .venv/bin/python3 -c "
import json
with open('data/processed/labels/{phase}_gold_manifest.jsonl') as f:
    for line in f:
        row = json.loads(line)
        if row['question_id'] == '{question_id}':
            print(json.dumps(row, indent=2))
            break
"

From the row you get: question_text, answer_type, gold_answer, gold_primary_pages, bucket, notes.

Organizer's Grounding Principle

The organizer's rule: "What pages would you show someone to justify your answer?" — the minimal set needed to support it at the scope asked by the question.

"Minimal" is relative to the claim being proved, not to a single document. You do NOT need every occurrence of a fact, but you DO need every page required to justify the answer at the full scope referenced in the question. If the question names two cases, two laws, or two entities, the grounding should reflect both — even if one alone logically suffices for the answer. For multi-case questions: "cite pages from ALL relevant cases. The grounding should reflect the full scope referenced in the question."

Apply this principle throughout the audit. A page set that logically proves the answer but doesn't cover the full scope of the question is under-grounded.

Audit Method

Step 1: Understand what the question needs

Read the question and answer. What specific information is required to produce this answer? Also identify the full scope — which cases, laws, or entities does the question reference? All of them need grounding.

• Comparison / cross-case questions ("common party in X and Y?", "Is ENF based on ARB?"): needs pages from ALL named cases — even if the answer is false or only one case's page logically suffices
• Law extraction ("Under Article X, is Y?"): needs the page with the article text
• Case data ("Date of Issue?"): needs the page with the formal date
• Case outcome ("What was the result?"): needs the order/disposition page

Step 2: Read cited pages and verify evidence placement

Read each cited page. For EVERY cited page, explicitly check:

• What evidence is on this exact page? (judge name, party name, article text, date, etc.)
• Is the evidence the answer needs actually HERE on this page number? If the cited page is p1 but the judge/party/date is on p2, that is a wrong_page issue — flag it immediately.

Do NOT assume the labeler cited the correct page number. The most common error is citing page 1 when the evidence is on page 2 (or vice versa). Write out what you found on each cited page before proceeding.

Then search independently — you cannot find missing pages by only looking at what's already cited.

You have 3 search tools. All bash commands must source the env first: cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a

Grep (exact text match — case numbers, article refs, party names):

Grep pattern="case_number" path="artifacts/runtime/{phase}/pages/"
Grep pattern="Article X" path="artifacts/runtime/{phase}/pages/"

FTS search (keyword + entity/article lookups, free, fast): Searches page text, entity names, aliases, summaries, article refs. Case number and article ref lookups get 3x boost.

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search_fts_and_lookups('{query}', top_k=15):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

Hybrid search (FTS + dense vectors, use if grep/FTS miss):

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search('{query}', top_k=15):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

Use your judgment on which tools and queries to use. Read any promising pages your searches surface that aren't already in the gold set.

Step 3: Systematically check for missing pages

Now that you've read the cited pages and done independent searches, check each scenario:

A. Multi-document questions (critical — apply the organizer's scope rule):

• Count how many cases/laws/entities the question names. Are pages from ALL of them in the gold set?
• Even if the answer is boolean false and one document's page logically proves it, the organizer expects grounding from all named documents.
• Do the search results contain pages from documents NOT in the gold set?
• For overlap questions: is the evidence page from EACH case document cited?
• For multi-sub-case questions: each sub-case may be a separate doc — are all covered?

B. Article/law questions:

• Find the exact article on the cited page. Does it start and end on the same page?
• If it continues on the next page → both pages must be cited
• "Subject to Article Y" or "notwithstanding Article Z" → did the search results or grep surface that article's page? Usually the cross-reference page is NOT part of the gold set unless the answer itself depends on text found only there.
• Defined terms → check if the definitions page appeared in search results

C. Date/name/case data questions:

• The search results may have surfaced both page 1 (header) and page 2 (order). Which is cited, which should be?
• Date of Issue → signature block page (p2-3), NOT header page (p1)
• Party names → page where the name appears in the case caption
• For ARB judge-overlap questions, treat arbitrators or tribunal members named in the ARB matter as the relevant judge-equivalent decision-makers. Do NOT count registrars or clerks.

D. Free-text grounding alignment: For free-text answers, go phrase by phrase through gold_answer:

• Find exact supporting text on a cited page for each phrase
• Flag any phrase with no page support
• Check if a page from the search results would provide the missing support — if so, recommend adding it

E. Absence-based answers:

• If the answer depends on absence from the corpus, or on the absence of the requested concept from a specific law/case file, gold_primary_pages = [] may be the correct gold set.
• Do NOT add title pages, table-of-contents pages, or scope pages just because they are related. Add a page only if it directly establishes the answer the judge should see.
• For boolean false, distinguish:
  • false from a specific provision or page-backed fact → cite that page
  • false from absence-based reasoning → empty gold pages can be correct

F. Off-by-one errors:

• Re-read each cited page and verify the quoted content is actually there
• Check page N-1 and N+1 if content seems shifted

Step 4: Check for unnecessary pages

Apply the strict necessity test to every cited page: "If I remove this page, can the answer still be fully verified from the remaining pages?" If yes, recommend removal.

Common unnecessary pages to catch:

• Title/cover pages when the answer content is entirely on an interior page
• A second document's page when one document's page alone proves the answer
• Pages that provide "context" (document identification) but no answer-relevant content
• Duplicate coverage — same info on two pages, only one needed

Step 5: Semantic ambiguity flag

If the question contains a deictic referent ("these Regulations", "this Law", "the Act") without naming a specific document:

• Check whether the labeler resolved the ambiguity correctly
• If multiple plausible referents exist and produce different answers → emit ambiguous_referent flag in notes for manual review
• Do NOT auto-relabel — just flag it. The orchestrator decides whether to relabel.

Output

Where to write

Mode	Audit JSON	Fix row?
row_path provided	artifacts/labeling/audits/{question_id}.json	Yes — update gold_primary_pages in row_path in-place
No row_path	artifacts/skill-calibration/verifier-grounding/{question_id}.json	No — audit only, don't modify the manifest

When fixing in-place: read the JSON from row_path, replace gold_primary_pages with suggested_pages, write back to the same file. Only do this when verdict is NOT correct.

Audit JSON shape:

{
  "question_id": "full_64char_hex_id",
  "question_text": "...",
  "answer_type": "boolean",
  "gold_answer": false,
  "audit_result": {
    "verdict": "correct | needs_more | too_many | wrong_page",
    "confidence": "high | medium | low",
    "current_pages": [{"doc_id": "...", "page_numbers": [1]}],
    "suggested_pages": [{"doc_id": "...", "page_numbers": [1, 
     
      
         
         
         
      
    
     
     
      
         
         
         
      
    
     
       
         
      
    
     
  
   

Verdict definitions

• correct — current pages are the right minimal set
• needs_more — missing one or more pages (the critical failure mode)
• too_many — some pages are unnecessary (mild penalty)
• wrong_page — a cited page doesn't contain relevant evidence (should be replaced)

Rules

• The answer is correct. Do NOT question it. Only audit pages.
• Read ALL cited pages before reaching a verdict.
• For each cited page, verify the evidence is on THAT EXACT page number — not an adjacent one. If you find evidence on page N+1 but the labeler cited page N, the verdict is wrong_page, not correct. This is the single most common verifier error.
• Grep independently — don't trust only the cited pages.
• Quote exact text from pages to justify every decision.
• Use full 64-char doc_id everywhere.
• Default to suggesting MORE pages when uncertain.
• Never substitute your own findings for the input. If your analysis shows the right page is p2 but the input says p1, flag the mismatch — don't silently report "correct".

References

• docs/evals/eval-plan.md § "Page-level grounding" — all grounding rules
• docs/gold-labeling.md — skill inventory, workflow, and publish-safety notes