verifier-ft-quality

name	verifier-ft-quality
description	Score a free-text gold answer against the 5 LLM judge criteria (correctness, completeness, grounding, confidence calibration, clarity/relevance) and suggest improvements. Use this skill after labeling free-text questions to catch quality issues before they cost points. Produces 0/0.5/1 scores per criterion and a suggested improved answer when any criterion is below 1.

Free-Text Quality Verifier

You evaluate a labeled free-text answer against the same criteria the competition's LLM judge uses. The judge sees: (1) the question, (2) a hidden reference answer, (3) our answer, (4) our cited pages as retrieved context.

We score 0.807 on assistant quality (29 failures out of 150 criterion checks on 30 FT questions). This skill finds and fixes those failures before submission.

Input

• question_id — the question to audit
• phase — warmup or final
• row_path (optional) — path to a file containing the JSON row. If provided, read from this file instead of the manifest.

Reading the row

If row_path is provided, read the row from that file:

cd "$(git rev-parse --show-toplevel)" && cat {row_path}

Otherwise, read the labeled row from the manifest (same as verifier-grounding).

Then read ALL pages in gold_primary_pages — these are the only pages the judge sees as context. Do NOT run an independent search — page completeness is handled by the verifier-grounding skill. This skill focuses purely on answer quality given the current page set.

Scoring Method

For each criterion, score 0 / 0.5 / 1:

1. CORRECTNESS (does the answer contain the right information?)

• What would the reference answer most likely say? (Infer from question + gold pages)
• Does our answer match that?
• Any factual errors — wrong numbers, wrong names, wrong dates?
• Score 0 if fundamentally wrong, 0.5 if partially correct, 1 if correct

2. COMPLETENESS (does the answer cover all key points?)

• List EVERY operative fact on the gold pages related to the question
• Which ones does our answer include? Which does it miss?
• The reference is also ~280 chars — it can't include everything. What would it prioritize?
• Enumeration exhaustiveness: For "Which laws...", "What cases...", "List all..." questions, the answer MUST name every item. A partial list with "and others" or "over a dozen more" is a completeness failure. If the gold_answer is non-exhaustive for an enumeration question, score 0.5 and rewrite with the full list (check reference_free_text for the complete set).
• Score 0 if misses the main point, 0.5 if covers some but not key facts, 1 if covers what matters

3. GROUNDING (is every word supported by the gold pages?)

This is the most important criterion and the one we fail most often.

Go through the answer phrase by phrase:

• For each phrase, find the EXACT supporting text on a gold page
• "Law No. 2 of 2018" but page says "this Law" → UNGROUNDED
• "shall" but page says "must" → mismatch
• Past tense when source uses present tense → mismatch
• Full entity name when page uses abbreviation → ungrounded

List every phrase with no support on the gold pages.

Score 0 if multiple ungrounded phrases, 0.5 if minor mismatches, 1 if fully grounded.

4. CONFIDENCE CALIBRATION (is the tone appropriate?)

• Court orders → definitive ("The application was dismissed")
• Uncertain provisions → hedged ("Based on the available text, ...")
• Unanswerable → clear denial, no speculation
• If gold_primary_pages = [], the best answer is usually a short absence-only statement. Extra legal background or explanatory "why" is usually ungrounded.
• Score 0 if tone completely wrong, 0.5 if slightly off, 1 if appropriate

5. CLARITY & RELEVANCE (clear, concise, on-point?)

• Does it answer the question directly?
• Within ~280 characters?
• No filler, no unnecessary preamble?
• Score 0 if irrelevant, 0.5 if verbose/indirect, 1 if clear and direct

Suggested Improvement

When any criterion scores below 1, write a suggested improved answer that:

1. Uses ONLY words and phrases that appear on the gold pages
2. Covers as many operative facts as possible within 280 chars
3. Directly answers the question
4. Does NOT introduce information not on the gold pages — even if factually correct

If gold_primary_pages = [], the suggested answer must stay a minimal meta-statement about absence. Do NOT add uncited explanations about DIFC law, court structure, or why the premise is wrong.

If adding a page to gold_primary_pages would allow a better answer (e.g., title page for the law's full name), note which page and why.

Output

Where to write

Mode	Audit JSON	Fix row?
row_path provided	artifacts/labeling/audits/ft-quality-{question_id}.json	Yes — update gold_answer in row_path in-place if suggested_answer improves score
No row_path	artifacts/skill-calibration/verifier-ft-quality/{question_id}.json	No — audit only, don't modify the manifest

When fixing in-place: read the JSON from row_path, replace gold_answer with suggested_answer, write back to the same file. Only do this when total_score < 5.

Audit JSON shape:

{
  "question_id": "...",
  "question_text": "...",
  "original_answer": "the gold_answer from manifest",
  "char_count": 245,
  "scores": {
    "correctness": 1,
    "completeness": 0.5,
    "grounding": 0,
    "confidence_calibration": 1,
    "clarity_relevance": 1
  },
  "total_score": 3.5,
  "grounding_issues": [
    {
      "phrase": "Law No. 2 of 2018",
      "issue": "gold page only says 'this Law'",
       
    
  
   
    
  
   
     
     
     
     
     
  
   
   
   
         
  
   

Semantic Flags

If you notice the question has a deictic referent ("these Regulations", "this Law") and the answer resolves it to a specific document, note ambiguous_referent in the audit. This is informational for manual review — do not auto-penalize.

Rules

• Audit gold_answer, not pipeline_answer — you're checking the gold label quality
• Read ALL gold pages before scoring
• The grounding check is the highest-priority criterion — every phrase must be on a gold page
• For unanswerable questions with empty gold pages: any factual claim is ungrounded by definition
• 280 chars is a soft target — don't sacrifice grounded completeness to hit it exactly
• For adversarial/unanswerable rows with empty gold pages, prefer short refusals like "The corpus does not contain information about X." Avoid explanatory refusals unless the explanation is itself grounded on cited pages.

References

• docs/evals/eval-plan.md § "Free-text grounding alignment" — common failure patterns
• docs/gold-labeling.md — skill inventory, workflow, and publish-safety notes