labeler-ft

name

labeler-ft

description

Label a free-text legal RAG question by searching the DIFC legal corpus and producing a gold manifest row with a grounded answer and minimal citation pages. Use this skill whenever you need to create gold labels for free_text questions — warm-up calibration, final phase labeling, or re-auditing. The critical challenge is grounding alignment — every phrase in the answer must be verifiable against the cited pages.

Free-Text Question Labeler

You are producing a gold-standard label for a free-text legal RAG question. Unlike deterministic questions where the answer is exact, free-text answers are scored by an LLM judge that checks your answer against a hidden reference answer using ONLY your cited pages as context.

This means: if your answer says something that's true but NOT on your cited pages, the judge marks it as ungrounded. Grounding alignment is the #1 priority.

Input

• question_id — full 64-char hex ID
• question_text — the question
• phase — warmup or final
• output_path (optional) — write the JSON row to this file instead of the manifest

Corpus pages: artifacts/runtime/{phase}/pages/{doc_id}/page-NNN.md Retrieval DB: artifacts/runtime/{phase}/retrieval.sqlite3

Search Tools

Same 3 tools as deterministic labeling. Use your judgment — look at the question, figure out what terms to search for, iterate.

All bash commands must source the env first: cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a

1. Grep (exact text match)

Grep pattern="case number or article ref" path="artifacts/runtime/{phase}/pages/"

2. FTS search (keyword + entity/article lookups, free, fast)

Searches page text, entity names, aliases, summaries, article refs. Also does case number and article ref → page lookups with 3x boost.

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search_fts_and_lookups('{query}', top_k=30):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

3. Hybrid search (FTS + dense vectors)

Adds embedding vectors. Uses one API call. Use when FTS + grep aren't enough.

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search('{query}', top_k=30):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

How to search effectively

• Always run at least 2 search methods. Don't stop after grep finds something — also run FTS.
• Start with grep for specific terms, then FTS for broader queries, hybrid if vocabulary mismatch.
• Read broadly — read the top 5-10 results. The verifier will trim unnecessary pages; your job is to not miss any.
• For free-text: pay attention to exact vocabulary on the pages you read. You'll need to use those exact words in your answer.
• Always read adjacent pages (N-1, N+1) for every page you plan to cite — content spans page boundaries frequently.
• Search with multiple query formulations — different terms surface different pages.

Deictic Referent Handling

Same rule as deterministic labeling:

1. Strong anchor exists (e.g., "these Regulations" + "Law of Security") → infer, search matching documents, answer if all plausible matches converge.
2. No anchor, all plausible matches agree → infer and answer.
3. No anchor, conflicting plausible answers → label unanswerable.

Try to resolve first. Only fall back to unanswerable when plausible referents produce genuinely different answers.

Enumeration Exhaustiveness

For questions asking "Which laws...", "What cases...", "List all..." — the answer MUST be exhaustive. Do not write "X, Y, Z, and over a dozen others." Name every item found. If the list is long:

• Put the complete list in gold_answer (use compact format)
• Also put the full list in reference_free_text as a checklist

A non-exhaustive answer to an enumeration question will fail the completeness criterion.

Writing the Answer

The judge's perspective

The LLM judge sees:

1. The question
2. A hidden reference answer (written by the organizer)
3. Your answer
4. Your cited pages (gold_primary_pages) as "retrieved context"

It scores 5 binary criteria (each 0 or 1):

1. CORRECTNESS — factually right given the evidence
2. COMPLETENESS — covers all key operative facts (for enumerations, ALL items named)
3. GROUNDING — every phrase supported by cited pages (most important)
4. CONFIDENCE CALIBRATION — tone matches certainty level
5. CLARITY & RELEVANCE — direct, concise, answers the question asked

Step 1: Find all relevant facts on the source pages

Before writing anything, list every operative fact on the pages you found that relates to the question.

Step 2: Draft using source vocabulary

Write 1–3 sentences. Aim for ~280 characters but never sacrifice correctness, completeness, or grounding for brevity. The critical rule: use the exact words and phrases from the source pages.

Common grounding failures to avoid:

• Writing "Law No. 2 of 2018" when the page says "this Law" → ungrounded
• Writing "shall" when the page says "must" → mismatch
• Writing "was refused" when the page says "is refused" → tense mismatch
• Adding full entity names when the page only uses abbreviations → ungrounded
• Paraphrasing "not exceeding" as "maximum" → ungrounded

Step 3: Verify every phrase

Go through your draft phrase by phrase. For each phrase, find the exact supporting text on one of your cited pages. If a phrase has no support:

• Rewrite it using vocabulary from the cited pages, OR
• Add the page that contains the supporting text to gold_primary_pages

Step 4: Check completeness

What would the reference answer likely say? Make sure your answer covers the key operative facts. The judge checks completeness — missing a key point costs a criterion.

Optimize for grounded completeness first, then tighten wording if possible.

Step 5: Calibrate confidence tone

• Court orders, clear provisions → definitive tone ("The application was dismissed")
• Uncertain or partial evidence → hedged ("Based on the available documents, ...")
• Unanswerable → short absence-only statement ("The corpus does not contain information about X")
• If gold_primary_pages = [], do NOT add explanatory legal background, jurisdiction facts, or "why" context. With empty gold pages, those extra claims are ungrounded.

Handling Unanswerable Questions

For free-text questions where the answer isn't in the corpus:

• gold_answer: a natural-language refusal statement (NOT null)
  • Good example: "The corpus does not contain information regarding jury trials."
  • Bad example: "The corpus does not contain information regarding jury trials, as DIFC courts do not use a jury system."
• gold_primary_pages: [] (empty — the answer depends on absence)
• answerability: "unanswerable"

For adversarial or unsupported free-text questions with empty gold pages:

• Keep the answer as a minimal meta-statement about absence only
• Do NOT explain the DIFC legal system, infer hidden facts, or justify the refusal with uncited background knowledge
• If you want to preserve the extra context for humans, put it in notes, not gold_answer

For questions where the answer is partially available:

• Answer what you can from the source pages
• Note limitations
• answerability: "ambiguous"

Grounding Page Selection

Same principles as deterministic labeling, with one addition:

Organizer's grounding rule

The organizer's principle: "What pages would you show someone to justify your answer?" — the minimal set needed to support it at the scope asked by the question.

"Minimal" is relative to the claim being proved, not to a single document. You do NOT need every occurrence of a fact, but you DO need every page required to justify the answer at the full scope referenced in the question. If the question names two cases, two laws, or two entities, the grounding should reflect both — even if one alone logically suffices. For multi-case questions, cite pages from ALL relevant cases.

Phrase-level grounding alignment

Every factual claim in your answer must be on a cited page. If your answer mentions a law name, that law name must appear on one of the gold_primary_pages. If it doesn't, either:

1. Remove the law name from your answer and use "this Law" (matching the page), or
2. Add the title page (where the full name appears) to gold_primary_pages

This is the phrase-level grounding alignment check — the single most important quality gate.

Include generously, verify later

Cite every page that could ground any part of your answer. When in doubt, include the page — the grounding verifier will trim unnecessary ones later. Missing a page costs ~6x more than an extra page (F-beta β=2.5).

Strict necessity test

For each page, ask: "If I remove this page, can the answer still be fully verified?" Drop pages that only provide context (title pages, headers) unless the answer text references content on that page.

Multi-doc phrase grounding

When citing pages from many documents (e.g., "Which laws mention X?"), remember: if your answer names a specific law or document, that name must appear on at least one cited page. This is just the standard phrase-level grounding rule — don't add extra title pages for "identification", only add them if the answer text references content on that page.

Output

Where to write

Mode	Write to
output_path provided	Write the JSON row (single line) to that file
No output_path	Write to {phase}_gold_manifest.jsonl using update-manifest-row (see below)

Same JSON row shape as deterministic labels, but with reference_free_text filled:

{
  "question_id": "full_64char_hex_id",
  "question_text": "...",
  "answer_type": "free_text",
  "bucket": "case_outcome_summaries",
  "answerability": "answerable",
  "gold_answer": "The application was dismissed. The court found no grounds to grant the relief sought by the claimant.",
  "gold_primary_pages": [
    {"doc_id": "full_64char_hex_id", "page_numbers": [2, 12]}
  ],
  "gold_support_pages": [],
  "reference_free_text": "Key facts: application dismissed, no grounds for relief, claimant's case rejected.",
  "notes": 
   
   

reference_free_text

Write the key facts that a good answer should mention. This helps later verification — it's a checklist, not a prose answer.

For unanswerable/adversarial questions with gold_primary_pages = [], leave reference_free_text empty unless there is a very specific checklist that does NOT add factual claims beyond absence.

Saving to manifest (only when no output_path)

cd "$(git rev-parse --show-toplevel)" && \
PYTHONPATH=src .venv/bin/python -m agentic_legal_rag update-manifest-row \
  --path "data/processed/labels/{phase}_gold_manifest.jsonl" \
  --question-id "{qid}" \
  --json '{"bucket":"...","answerability":"...","gold_answer":"...","gold_primary_pages":[...],"gold_support_pages":[],"reference_free_text":"...","notes":"...","label_confidence":"...","review_status":"drafted"}'

References

• docs/evals/eval-plan.md § "Free-text grounding alignment" — common failure patterns
• docs/evals/eval-plan.md § "Grounding Principles" — page-level rules
• docs/evals/gold-dataset-spec.md — row shape and answer rules