audit-gold

name	audit-gold
description	Audit a gold manifest row end-to-end — verify answer correctness, grounding pages, and free-text quality. Reads from manifest, searches corpus independently, and updates the row in-place (review_status → audited or flagged). Use this skill for post-labeling verification of any question type.

Gold Label Auditor

You audit a single gold manifest row for a legal RAG competition over a DIFC legal corpus. Unlike the separate verifier skills, you check EVERYTHING in one pass: answer correctness, grounding pages, and (for free_text) answer quality.

Your job: find errors before the competition scores them. Be adversarial — assume the labeler made mistakes.

Competition Scoring Context

Total = (0.7 * deterministic_accuracy + 0.3 * assistant_quality) * GROUNDING * telemetry * ttft_factor

Grounding is a MULTIPLIER — perfect answers score 0 if grounding is wrong.

Grounding uses F-beta (β=2.5): recall-weighted → missing a gold page costs ~6× more than an extra page.

Answer type rules

• boolean: JSON true/false (exact match)
• number: numeric value only, ±1% tolerance
• date: YYYY-MM-DD exact match
• name: normalized exact match (strip().lower())
• names: JSON array of strings, Jaccard similarity over normalized sets
• free_text: scored by LLM judge on 5 criteria, each 0 or 1. The judge sees the question, a hidden reference answer, our answer, and our cited pages. Answer should be 1–3 sentences (~280 chars), but never sacrifice correctness/completeness/grounding for brevity.
  1. CORRECTNESS — factually right given the evidence
  2. COMPLETENESS — covers all key operative facts (for enumerations, ALL items named)
  3. GROUNDING — every phrase supported by cited pages (most important)
  4. CONFIDENCE CALIBRATION — tone matches certainty (court orders → definitive, unanswerable → minimal denial)
  5. CLARITY & RELEVANCE — direct, concise, answers the question asked

Organizer grounding principles

• "What pages would you show someone to justify your answer?" — the minimal set at the FULL SCOPE of the question.
• If the question names two cases, two laws, or two entities → grounding must reflect ALL of them, even if one alone logically proves the answer.
• For multi-case comparison/overlap questions: cite pages from ALL relevant cases.
• When a date appears on multiple pages, prefer the page where the label explicitly matches the question wording (e.g., "Date of issue:" label).
• If the corpus contains no information → pages should be empty ([]).
• For boolean false from absence → empty pages. For boolean false from a specific provision → cite that provision's page.
• Both empty predicted + empty gold → grounding = 1.0.

Input

• question_id — full 64-char hex ID (or unique prefix)
• phase — warmup or final (default final)

Step 0: Read the Row

cd "$(git rev-parse --show-toplevel)" && \
python3 -c "
import json, sys
qid_prefix = '{question_id}'
with open('data/processed/labels/{phase}_gold_manifest.jsonl') as f:
    for line in f:
        row = json.loads(line)
        if row['question_id'].startswith(qid_prefix):
            print(json.dumps(row, indent=2))
            sys.exit(0)
print('NOT FOUND', file=sys.stderr)
sys.exit(1)
"

From the row you have: question_id, question_text, answer_type, bucket, answerability, gold_answer, gold_primary_pages, notes, reference_free_text.

Step 1: Read ALL Cited Pages

For every entry in gold_primary_pages, read the page markdown:

Read file_path: artifacts/runtime/{phase}/pages/{doc_id}/page-{NNN}.md

where NNN is zero-padded to 3 digits. Read ALL pages before proceeding. Write down what evidence each page contains — you'll need this for the grounding check.

Step 2: Verify Answer Correctness

Based ONLY on the page contents you just read plus independent search:

For deterministic types (boolean, number, date, name, names):

• Is the gold_answer factually correct given the evidence?
• For boolean: does the evidence support true or false?
• For number: is the value exactly right (count manually if needed)?
• For date: is the format YYYY-MM-DD and the date correct?
• For name/names: is the extracted value correct and normalized?

For free_text:

• Is the answer accurate and complete given the evidence?
• Does it answer the question directly?

For unanswerable:

• Is it truly absent from the corpus? (Quick grep/FTS to confirm)

If the answer seems wrong, search independently (Step 3) BEFORE concluding — you might find the answer on a different page.

Step 3: Independent Search for Evidence

Search the corpus independently. Do NOT rely only on the cited pages — the labeler may have missed pages or cited the wrong ones.

All bash commands must source the env first: cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a

Grep (exact text match — case numbers, article refs, party names):

Grep pattern="case_number_or_article" path="artifacts/runtime/{phase}/pages/"

FTS search (keyword + entity/article lookups):

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search_fts_and_lookups('{query}', top_k=15):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

Hybrid search (FTS + dense vectors — use if grep/FTS miss):

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search('{query}', top_k=15):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

Use targeted queries. For multi-case questions, search for each case number separately. For article questions, grep the article reference. Read any promising pages your searches surface that aren't in the gold set.

Step 4: Grounding Audit

4a. Page correctness — for EACH cited page:

1. Is the evidence actually on THIS page number? The most common error is off-by-one (cited p1 but evidence is on p2). Re-read the page you loaded in Step 1 and confirm the specific text the answer relies on is there.
2. Strict necessity test: If you remove this page, can the answer still be fully verified from the remaining pages? If yes → flag as unnecessary.

4b. Missing pages — THIS IS CRITICAL

Check each scenario:

Multi-document questions (apply organizer scope rule):

• Count how many cases/laws/entities the question names. Are pages from ALL of them in the gold set?
• Even if the answer is boolean false, the organizer expects grounding from all named documents.

Article/law questions:

• Does the article span multiple pages? If so, all pages must be cited.
• Cross-references ("Subject to Article Y") → usually NOT needed unless the answer text itself depends on that article.

Date/name/case data questions:

• Did the labeler cite the right page? (Date of Issue → signature block page, not header)
• For ARB questions: arbitrators/tribunal members are the relevant judge-equivalent, not registrars/clerks.

Off-by-one errors:

• Check page N-1 and N+1 for each cited page if anything looks shifted.

4c. Free-text phrase-level grounding (free_text only):

Go through gold_answer phrase by phrase:

• Find exact supporting text on a cited page for EACH phrase
• Flag: vocabulary mismatches ("Law No. 2 of 2018" vs page says "this Law"), tense mismatches, abbreviation expansions, paraphrasing not on the page
• Every phrase must be traceable to a specific page

Step 5: Free-Text Quality Check (free_text only)

Score each of the 5 criteria 0 / 0.5 / 1 (same criteria listed in the answer type rules above):

1. CORRECTNESS — does the answer contain the right information?
2. COMPLETENESS — does it cover the key operative facts? For enumeration questions ("Which laws...", "List all..."), ALL items must be named — no "and others" summaries. Never sacrifice completeness for brevity.
3. GROUNDING — is every word supported by the gold pages? (most important criterion)
4. CONFIDENCE CALIBRATION — tone matches certainty level? Court orders → definitive. Unanswerable → minimal denial.
5. CLARITY & RELEVANCE — direct, concise (1–3 sentences), answers the question?

If any criterion scores below 1, write a suggested improved answer using ONLY words from the gold pages. Keep it to 1–3 sentences. Prioritize correctness, completeness, and grounding over brevity.

Step 6: Produce Verdict and Write Audit Artifact

Verdict

• correct — answer and pages are right (or issues are trivial enough to not affect scoring)
• needs_fix — answer wrong, pages wrong, or free-text quality issues that would lose points

Write audit artifact

IMPORTANT: Do NOT modify the manifest file directly. Multiple audit agents may run in parallel, and concurrent writes corrupt the file. Instead, write your findings to an audit artifact. The orchestrator will apply manifest updates sequentially after all agents finish.

mkdir -p artifacts/labeling/audits

Write to artifacts/labeling/audits/audit-{question_id_first_8}.json:

{
  "question_id": "full_64char_id",
  "question_text": "...",
  "answer_type": "...",
  "verdict": "correct | needs_fix",
  "answer_audit": {
    "status": "correct | wrong",
    "issue": "description if wrong",
    "original_answer": "...",
    "suggested_answer": "... (only if wrong)"
  },
  "grounding_audit": {
    "status": "correct | needs_more | too_many | wrong_page",
    "current_pages": [...],
    "suggested_pages": [...],
    "missing_pages":    N  
        N  
      
  
   
              
     
          
     
  
   
     
     
     
     
  
   

For deterministic questions, omit the ft_quality field.

The manifest_update field contains the exact changes the orchestrator should apply:

If verdict is correct — only review_status is needed:

"manifest_update": {"review_status": "audited"}

If verdict is needs_fix — include all fields that need changing:

"manifest_update": {
  "review_status": "audited",
  "gold_answer": "SUGGESTED_ANSWER",
  "gold_primary_pages": [...],
  "notes": "UPDATED_NOTES"
}

Only include fields that actually changed (plus review_status always). When fixing, the notes value should append what the audit found. Format: "EXISTING_NOTES [AUDIT] fixed: description. Original: ...".

Rules

• Read ALL cited pages before reaching any verdict.
• Search independently — don't trust the labeler's work.
• Quote exact text from pages to justify every finding.
• For each cited page, verify evidence is on THAT EXACT page number.
• Default to suggesting MORE pages when uncertain (recall >> precision in F-beta β=2.5).
• Do NOT add title/cover/ToC pages unless they contain answer-relevant content.
• For unanswerable questions with [] pages: skip grounding audit, just confirm absence.
• Use full 64-char doc_id everywhere.
• The gold_support_pages field is always [] — never populate it.
• NEVER write to the manifest JSONL file directly. Only write the audit artifact JSON.

Print summary at the end

Print a one-line summary:

AUDIT {question_id_first_8} [{answer_type}] → {verdict}: {one-line description}

Examples:

AUDIT {qid} [name] → correct: answer and pages verified
AUDIT {qid} [free_text] → needs_fix: missing page {doc_id} p12, rewrote answer for grounding