| name | adjudication-sheets |
| description | Build human adjudication / hand-labeling sheets from LLM-pipeline data without
evidence truncation. Use when: (1) preparing a CSV/Excel sheet for a human to
rule on cases an LLM classifier or rater panel judged, (2) a labeler reports
"there is no information to label from" or cells look empty in Excel,
(3) excerpt columns cluster at one exact length (e.g. all 1,500 chars — a hard
truncation cap). Covers: full rating-basis recovery, Excel 32,767-char cell cap,
multi-line CSV mangling, ruling dropdowns, companion text files.
|
| author | Claude Code |
| version | 1.1.0 |
| date | 2026-07-08T00:00:00.000Z |
Human Adjudication Sheets from LLM-Pipeline Data
Problem
Adjudication sheets built from pipeline intermediates tend to carry truncated
"excerpt" columns (previews made for machine diffing, not human judging). A human
asked to rule "does this text show X?" on a fragment produces invalid rulings:
the decisive sentence may sit past the cutoff. Separately, multi-line text in CSV
cells renders as broken rows/empty cells in Excel, so the labeler reports "there
is no data here" even when the column is populated.
Context / Trigger Conditions
- A labeler says the sheet has "no information" — check for embedded newlines in CSV fields first.
- Excerpt lengths cluster at an exact value (all 1,204 / 1,500 / 800 chars) or end with "[…]" — that is a hard cap, not natural length.
- The ruling question is "does the record/text show X?" — the ruling basis must be byte-identical to what the classifier/raters saw, or the ruling grades a different object.
Solution
- Trace the true rating basis from the scoring code, not from intermediates. Find the line where text is passed to the model (e.g.
prompt = ... + text) and reproduce that construction exactly. Verify: stored excerpt == full_text[:cap] (0 mismatches), or cache-key match against the LLM request cache.
- Deliver .xlsx, not .csv, for any sheet with multi-line text: wrap_text + frozen header + generous width on evidence columns; data-validation dropdown on the
ruling column with the exact allowed codes (e.g. E_pos/E_neg/unclear, or 0/1/2/unclear).
- Handle the Excel cell cap (32,767 chars) explicitly: if full text exceeds it, inline the first ~30k ending with a loud marker ("[CONTINUES — see companion file]") AND write the complete text to a per-case
.txt (header block with case metadata + full text); add a text_file path column. Write companion files for ALL cases regardless — they are the comfortable reading surface.
- Keep the original CSV untouched as the machine-readable artifact; the xlsx is the human ruling surface; harvest rulings from the xlsx.
Distribution:
text_file paths are project-relative and break the moment the sheet is emailed or copied out of the tree (the labeler asks "where are the attendant/companion files?"). If the labeler is not working inside the project folder, ship ONE zip containing the xlsx and the companion-texts folder side by side, with the instruction: extract together; the file for row X is <texts_folder>\<case_id>.txt next to the spreadsheet. Require the FILLED xlsx back and harvest from the returned copy, not the original.
- Tell the labeler the epistemics: rule from the provided record only; silence = negative; outside research (Google) goes in
ruling_notes, never the ruling — otherwise the labels leak information the classifier could never see.
Verification
- No excerpt column has >30% of rows at one exact length; no "[…]" markers remain.
pd.read_excel round-trip shows full lengths (compare min/median/max vs the old excerpts).- Dropdown rejects free-text entries; a saved test ruling survives reopen.
Example
Specialist Directors US, 2026-07-08: director sheet excerpts capped at ~1,200 chars (full dossiers up to 4,090); filing sheet capped at 1,500 chars while raters had scored the whole ~80k-char Item 1A — the PI caught both mid-sitting. Fix: src/director_v1/make_sitting_xlsx.py, tools/make_adjudication_v2_xlsx.py (full-text xlsx + 76 companion txts + dropdowns).
Notes
- Uniform-length clustering is the fastest tell; check it BEFORE handing any sheet to a human.
- Harvest gotcha: when comparing harvested rulings to prior labels, coerce BOTH sides to numeric first — Excel/pandas round-trips floats as "1.0" vs the dropdown's "1", so a string compare falsely flags every row as a disagreement.
- If the pipeline caches LLM requests content-addressed, verify the rebuilt basis against the cache rather than trusting a deterministic builder to have been stable.
