labeler-det

name

labeler-det

description

Label a deterministic legal RAG question (boolean, number, date, name, names) by independently searching the DIFC legal corpus and producing a gold manifest row with the correct answer and minimal grounding pages. Use this skill whenever you need to create or re-label gold rows for deterministic questions — warm-up calibration, final phase labeling, or re-auditing existing labels. Covers answer types boolean, number, date, name, names.

Deterministic Question Labeler

You are producing a gold-standard label for a legal RAG competition question. Your output will be compared against a hidden golden set of pages by the competition platform — getting the right answer is necessary but not sufficient; you must also identify the exact minimal pages that prove it.

Input

You receive these parameters (provided in the invocation prompt):

• question_id — full 64-char hex ID
• question_text — the question
• answer_type — one of: boolean, number, date, name, names
• phase — warmup or final (determines file paths)
• output_path (optional) — write the JSON row to this file instead of the manifest

Corpus pages: artifacts/runtime/{phase}/pages/{doc_id}/page-NNN.md Retrieval DB: artifacts/runtime/{phase}/retrieval.sqlite3

Search Tools

You have 3 search tools. Use your judgment on which to use and in what order — look at the question, figure out what terms to search for, and iterate. You do NOT need to read everything; be targeted.

All bash commands must source the env first: cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a

1. Grep (exact text match)

Use the built-in Grep tool. Best for case numbers, article references, party names, law titles — anything where you know the exact string.

Grep pattern="CFI 057/2025" path="artifacts/runtime/{phase}/pages/"
Grep pattern="Article 8\\(1\\)" path="artifacts/runtime/{phase}/pages/"
Grep pattern="Operating Law" path="artifacts/runtime/{phase}/pages/"

2. FTS search (keyword relevance + entity/article lookups, free, fast)

Searches across 5 FTS columns (search_text, entity_names, aliases, retrieval_summary, article_references) plus case number and article index lookups with 3x RRF boost. No API calls.

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search_fts_and_lookups('{query}', top_k=30):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

This searches: page text, entity names (parties, judges), aliases, per-page summaries, and article references. It also does direct case number → doc_id lookup and article ref → page lookup, both boosted 3x in fusion.

3. Hybrid search (FTS + dense vectors + scoped search)

Adds section-level and page-level embedding vectors to FTS. Uses one embedding API call. Use when FTS + grep aren't enough (vocabulary mismatch, semantic concepts).

cd "$(git rev-parse --show-toplevel)" && set -a && source .env && set +a && \
PYTHONPATH=src .venv/bin/python3 -c "
from agentic_legal_rag.retrieval.search import HybridSearcher
from agentic_legal_rag.enrichment.client import IsaacusClient
s = HybridSearcher('artifacts/runtime/{phase}/retrieval.sqlite3', IsaacusClient())
for r in s.search('{query}', top_k=30):
    print(f'{r.doc_id} p{r.page_number} score={r.rrf_score:.4f}')
"

How to search effectively

• Always run at least 2 search methods. Don't stop after grep finds something — also run FTS to catch what grep misses.
• Start with grep for specific terms (case numbers, article refs, party names). Instant and precise.
• Use FTS for broader keyword search — it picks up entity names, aliases, and article references automatically.
• Use hybrid when FTS misses (vocabulary mismatch, synonyms, concepts).
• Read broadly — read the top 5-10 results, not just 2-3. The verifier will trim unnecessary pages later; your job is to not miss any.
• Always read adjacent pages (N-1, N+1) for every page you plan to cite — content often spans page boundaries and off-by-one errors are common.
• Search with multiple query formulations — different terms, different phrasings. A single query rarely surfaces everything.

When to stop

• You've found the answer AND verified pages from all relevant documents are included
• For multi-document questions: you've confirmed coverage of EVERY document referenced or implied
• You've tried at least 2 search methods with multiple query formulations and found nothing → mark unsupported

Deictic Referent Handling

Questions may use "these Regulations", "this Law", "the Act" without naming a specific document. Apply this rule:

1. Strong anchor exists (e.g., "these Regulations" + "Law of Security" in the same question) → infer the referent, search for matching documents, and answer if all plausible matches converge on the same answer.
2. No anchor, but all plausible matches agree → infer and answer.
3. No anchor, and plausible matches produce conflicting answers → label unanswerable. This is the key test: conflicting plausible referents = unanswerable.

Do NOT default to unanswerable just because a referent is technically unanchored. Try to resolve it first.

How to Determine the Answer

boolean

Find the specific provision or fact. Return JSON true or false.

• "Is X permitted under Article Y?" → find the article, read what it says
• "Do cases X and Y share a common party?" → find parties of each case, compare sets
• "Judge" means titled Justices or Judges (e.g., "H.E. Justice X", "Chief Justice Y"). It does NOT include registrars, clerks, or other court officers — even if they issue orders or make judicial decisions. An "Assistant Registrar" is not a judge.
• ARB exception: if a judge-overlap question involves an ARB case, treat arbitrators or tribunal members named as the decision-makers in that ARB matter as the relevant "judge-equivalent" for overlap comparison. Still exclude registrars and clerks.
• Use false when the corpus lets you answer the yes/no question negatively, even if that negative depends on absence from a specific law or case file.
• Use null only when the question is genuinely unsupported by the corpus or asks for a concept outside the competition evidence surface with no document-anchored yes/no determination.
• Temporal impossibility questions ("Did ENF 2019 enforce ARB 2023?") should usually be false, not null.

number

Extract the numeric value only. No units, no currency symbols, no text. Return a JSON number.

date

Format as "YYYY-MM-DD". Key: "Date of Issue" is on the registrar's signature/order page (usually page 2-3), NOT the cover/header page.

name

One normalized string. Use the form as it appears in the document (don't invent abbreviations or expansions).

names

JSON list of strings. Order doesn't matter. Include all relevant entities.

null (unsupported)

If the question asks about something not in the corpus at all, answer is null.

Grounding Page Selection

This is where most scoring points are won or lost. The competition uses F-beta with β=2.5 — missing a page costs ~6x more than an extra page.

Core principle — organizer's grounding rule

The organizer's rule: "What pages would you show someone to justify your answer?" — the minimal set needed to support it at the scope asked by the question.

"Minimal" is relative to the claim being proved, not to a single document. You do NOT need every occurrence of a fact, but you DO need every page required to justify the answer at the full scope referenced in the question. If the question names two cases, two laws, or two entities, the grounding should reflect both — even if one alone logically suffices for the answer.

When in doubt, include the page — the recall penalty for missing one is catastrophic (F-beta β=2.5, missing costs ~6x more than an extra page).

Strict necessity test

For each page you plan to cite, ask: "If I remove this page, can the answer still be fully verified from the remaining pages?" If yes, drop it. Common unnecessary pages:

• Title/cover pages when the answer content is entirely on an interior page
• "Context" pages that identify the document but don't contain answer-relevant content

Exception — multi-case questions override this test. The organizer rule is: "cite pages from ALL relevant cases. The grounding should reflect the full scope referenced in the question." If a question names two cases (overlap, comparison, or cross-reference), cite evidence from EACH case — even when the answer is boolean false and one page logically suffices. Both pages are needed to ground the full scope of what the question asks about.

Rules by question type

Cross-case overlap ("Do X and Y share a party/judge?"):

• The page from EACH case document where the relevant evidence actually appears (parties in caption, judge name in heading)
• This is USUALLY page 1, but not always — some documents have a cover page on p1 with no judge/party names; the actual caption or judge heading may be on p2. Always verify by reading.
• Even when the answer is false, cite the evidence page from each case document
• For sub-cases within a parent case: each sub-case is often a separate document

Law/article extraction ("Under Article X, is Y true?"):

• The page containing the article text
• If the article continues on the next page → cite BOTH pages
• If the article says "subject to Article Z" → check whether the referenced article's page is needed
• If the article uses a defined term → consider whether the definitions page should be included

Date of Issue:

• The signature block / order page (usually page 2-3)
• NOT the header/cover page (page 1) — this is the single most common grounding error

Case outcome / court orders:

• The order/disposition section (often page 2 or near the end)
• Include both the order heading and the operative paragraphs if they span pages

Case party extraction:

• The page where the party name actually appears in the caption or heading
• Not just any page that mentions the party in passing

Case comparisons ("Which case has the earlier date / higher amount?"):

• Cite the page where the compared data actually appears (date, monetary amount, etc.)
• For monetary claims: the definitive figure may be on the claim form page or in the judgment's summary section — not just p1 or p2. Search for the actual number.
• Don't just cite p1 for case identification — the competition needs the page with the data being compared

Unanswerable / adversarial:

• gold_primary_pages = [] (empty list)

Boolean false from absence (no evidence in corpus):

• gold_primary_pages = []

Boolean false from a specific provision:

• Cite the page containing the provision that establishes the negative

Final page verification (MANDATORY)

Before finalizing your output, re-read EVERY page you plan to cite and confirm:

1. The specific evidence (judge name, party name, article text, date, amount) is actually ON that exact page number
2. If the evidence is NOT on the page you planned to cite, check adjacent pages (N-1, N+1) and cite the correct one instead
3. Write the exact quote from each cited page in your notes

This catches the most common labeling error: citing page 1 when the evidence is actually on page 2 (e.g., some documents have a cover page on p1 with no judge name — the judge heading is on p2).

Output

Where to write

Mode	Write to
output_path provided	Write the JSON row (single line) to that file
No output_path	Write to {phase}_gold_manifest.jsonl using update-manifest-row (see below)

JSON row shape

{
  "question_id": "full_64char_hex_id",
  "question_text": "...",
  "answer_type": "boolean",
  "bucket": "cross_case_overlap",
  "answerability": "answerable",
  "gold_answer": false,
  "gold_primary_pages": [
    {"doc_id": "full_64char_hex_id", "page_numbers": [1]}
  ],
  "gold_support_pages": [],
  "reference_free_text": "",
  "notes": "Exact quote: 'SCT 295/2025 — OLEXA v ODON' (p1 doc A). No shared parties.",
  "label_confidence": 
   

Bucket values

• law_and_document_extraction — article/law metadata, booleans about law text, dates, law numbers
• cross_case_overlap — shared parties, shared judges across cases
• case_comparisons — earlier date, higher claim value between cases
• case_outcome_summaries — court orders, application results, rulings
• case_party_extraction — defendant, claimants from case docs
• adversarial_or_unanswerable — premises not in DIFC corpus

Bucket consistency rules

• Case-level "Date of Issue" questions belong in law_and_document_extraction for consistency.
• Single-case judge-composition or judge-change questions belong in case_outcome_summaries, not cross_case_overlap.
• Law-vs-law comparison questions belong in law_and_document_extraction, not case_comparisons.

Saving to manifest (only when no output_path)

cd "$(git rev-parse --show-toplevel)" && \
PYTHONPATH=src .venv/bin/python -m agentic_legal_rag update-manifest-row \
  --path "data/processed/labels/{phase}_gold_manifest.jsonl" \
  --question-id "{qid}" \
  --json '{"bucket":"...","answerability":"...","gold_answer":...,"gold_primary_pages":[...],"gold_support_pages":[],"reference_free_text":"","notes":"...","label_confidence":"...","review_status":"drafted"}'

Notes field

The notes field is required. Include:

1. The exact quote from the source page that supports the answer
2. Any cross-references or ambiguity
3. Which search path found the evidence (helps debug failures later)

Confidence calibration

• high — found the answer directly on a specific page, no ambiguity
• medium — answer requires inference or combining information from multiple pages
• low — answer is plausible but evidence is indirect or partial

References

For deeper context on grounding rules and scoring, read:

• docs/evals/eval-plan.md § "Grounding Principles"
• docs/evals/gold-dataset-spec.md — canonical row shape
• docs/challenge/brief.md — competition scoring formula