eval-ft-answer

name	eval-ft-answer
description	Score a pipeline's free-text answer against the gold answer and gold pages on the 5 competition criteria (correctness, completeness, grounding, confidence calibration, clarity). Use this skill to evaluate answer quality for free_text questions before submission. Input is a question_id prefix; it reads the submission answer, gold answer, and gold pages automatically.

Free-Text Answer Evaluator

You score a pipeline-generated answer against the gold answer using the same 5 criteria the competition's LLM judge uses. The judge sees: (1) the question, (2) a hidden reference answer, (3) our answer, (4) our cited pages as retrieved context.

Our gold_answer is our best proxy for the organizer's hidden reference — it scores 0.84 on the platform, which appears to be a ceiling. Use it as the comparison target, but also verify facts independently against the gold pages. If the gold answer and gold pages disagree, trust the pages.

Input

• question_id — question ID or prefix (e.g. 436c12cc)

Step 1: Load Data

Gold manifest row

Run this bash command to extract the single matching row (replace {QID} with the question_id prefix):

grep '{QID}' data/processed/labels/warmup_gold_manifest.jsonl

Verify answer_type == "free_text". Extract: question_text, gold_answer (the comparison target), gold_primary_pages, bucket, answerability, notes.

Submission answer

Run this bash command to extract the matching answer:

python3 -c "
import json
with open('artifacts/submissions/warmup/submission.json') as f:
    sub = json.load(f)
for ans in sub['answers']:
    if ans['question_id'].startswith('{QID}'):
        print(json.dumps(ans, indent=2))
        break
"

Extract: answer (the submitted answer).

Step 2: Read Gold Pages

Gold pages live at: artifacts/runtime/warmup/pages/{doc_id}/page-{NNN}.md

Where NNN is the zero-padded 3-digit page number (e.g., page 1 → page-001.md, page 12 → page-012.md).

For each [doc_id, page_number] pair in gold_primary_pages, read the corresponding .md file. These are the ONLY pages the judge sees as context.

If gold_primary_pages is empty ([]), the question is unanswerable — skip page reading.

Print the submitted answer and the gold answer before scoring.

Step 3: Score on 5 Criteria (PASS / FAIL)

Use binary scoring (PASS/FAIL) matching the platform judge. Go through each criterion:

1. CORRECTNESS (does the submitted answer match what the gold answer says?)

• The gold answer tells you what the correct answer contains.
• Does the submitted answer contain the KEY information? Any factual errors?
• Wrong numbers, wrong names, wrong dates, wrong verbs?
• Common failure: answering a slightly different question, or getting a detail wrong.

2. COMPLETENESS (does the submitted answer cover what the gold answer covers?)

• List EVERY operative fact in the gold answer.
• Which ones does the submitted answer include? Which does it miss?
• For court orders: the gold answer likely lists ALL items from "IT IS HEREBY ORDERED" — ruling, costs, deadlines, conditions.
• For statutes: the gold answer includes article/section references.
• For multi-part questions: the gold answer addresses every part.
• Common failure: court order answers that miss operative items (costs, filing deadlines, interest). Law answers missing article references.
• Do not add fluff to chase completeness. Only facts needed for the answer.

3. GROUNDING (is every phrase in the submitted answer supported by the gold pages?)

This is the most critical criterion. Go phrase by phrase:

• For EACH phrase, find EXACT supporting text on a gold page. Quote it.
• The judge sees ONLY the gold pages as context. If the answer says something not on those pages, it's ungrounded — even if factually correct.
• Known failure patterns:
  • Law names not on the cited pages (page says "this Law", answer says "Employment Law 2019") → UNGROUNDED
  • Article/section labels mismatched ("Article 6" vs page says "6." or "section 6")
  • Paraphrases ("must" vs "shall", "empowers" vs "may make", "max" vs "not exceeding")
  • Tense ("was refused" vs source says "is refused")
  • Adding entity names the page doesn't mention (full party names, case citations)
  • Adding interpretive claims ("provisions apply retrospectively" when page just gives dates)
• For unanswerable questions with empty gold pages: ANY factual claim is ungrounded.

4. CONFIDENCE CALIBRATION (is the tone right?)

The evaluator penalizes BOTH directions:

• Clear answer in document → respond directly and confidently. DO NOT hedge ("it appears that", "based on the documents") when the source is definitive.
• Ambiguous/incomplete → explicitly acknowledge uncertainty.
• Not present in corpus → state clearly, no speculation.
• Citing the source ("Under Article 8...", "Article 16(1)(c) requires...") is NOT hedging — it is attribution.

5. CLARITY & RELEVANCE (clear, concise, direct?)

• Does it directly answer the specific question asked?
• Within ~280 characters? (soft target — judge prefers concise)
• No filler words or preamble?
• Lead with the direct answer, then supporting details.
• For court orders with multiple items: use semicolons or short numbered format.

Step 4: Identify Prompt Improvements

Do NOT write a suggested improved answer. Instead, identify what the pipeline prompt should change to fix the failures. Think about:

• Is the LLM too terse? (completeness failure) → prompt needs to emphasize including all operative items
• Is the LLM using wrong vocabulary? (grounding failure) → prompt needs stronger "use exact page words" instruction
• Is the LLM hedging on clear facts? (calibration failure) → prompt needs confidence guidance
• Is the LLM missing article references? (completeness failure) → prompt needs "always include article/section label" rule

Frame your feedback as prompt-level fixes, not per-answer rewrites.

Output

Print to stdout:

=== {question_id[:8]} ({bucket}, {answerable|unanswerable}) ===
Q: {question_text}
SUB ({char_count} chars): {submitted_answer}
GOLD: {gold_answer}

1. CORRECTNESS: PASS/FAIL
   {1-2 sentence reasoning}

2. COMPLETENESS: PASS/FAIL
   {reasoning — list what's missing if FAIL}

3. GROUNDING: PASS/FAIL
   {reasoning — list ungrounded phrases if FAIL}

4. CONFIDENCE CALIBRATION: PASS/FAIL
   {reasoning}

5. CLARITY & RELEVANCE: PASS/FAIL
   {reasoning}

SCORE: {X}/5
PROMPT FEEDBACK: {what the pipeline prompt should change to fix failures}

Also write JSON to artifacts/eval-ft-quality/{question_id[:8]}.json:

{
  "question_id": "...",
  "bucket": "...",
  "answerable": true,
  "submitted_answer": "...",
  "gold_answer": "...",
  "submitted_char_count": 48,
  "scores": {
    "correctness": "PASS",
    "completeness": "FAIL",
    "grounding": "PASS",
    "confidence_calibration": "PASS",
    "clarity_relevance": "PASS"
  },
  "pass_count": 4,
  "failures": ["completeness"
    
   
   
   

Rules

• You are scoring the SUBMISSION answer against the gold answer and gold pages
• Read ALL gold pages before scoring — grounding check requires actual page text
• Use PASS/FAIL (binary), not 0/0.5/1 — matching the platform judge
• The gold answer tells you what's correct; the gold pages tell you what's grounded
• A submitted answer can be correct but ungrounded (right fact, wrong words)
• A submitted answer can be grounded but incomplete (right words, missing facts)
• Do NOT generate an improved answer — generate prompt feedback instead