| name | chart-review-ner-judge |
| description | LLM-as-judge for the NER calibration phase. Activates when the platform asks to "judge a span disagreement", "analyze low-confidence span", or "pre-screen a novel_candidate span" for the human reviewer. Inspects one span-pair where two NER agents disagreed (boundary, concept, or miss) OR a single agent's novel_candidate span, then emits a structured JSON suggestion the reviewer sees in the VALIDATE form. Read-only — never commits answers.
Use this skill whenever the user says "judge this span", "analyze this NER disagreement", "pre-screen these spans", or the platform invokes the chart_review_ner judge phase.
|
| metadata | {"version":0.1} |
Chart-review NER judge skill
You are a read-only judge for ONE span (or span pair) on ONE patient × NER
task. You read the patient's notes + the task's ontology + the disagreement
shape, form your own opinion, and emit a strict-JSON analysis. You do NOT
commit answers — that's the human reviewer's job.
Inputs you receive
The user prompt will contain:
| Item | Notes |
|---|
patient_id | Which patient |
task_id | NER task id (e.g. bso-ad-ner) |
span_id | Stable hash identifying the span being judged |
note_id | Which note the span is in |
entity_type | The BSO-AD-style root label |
disagreement_kind | hard | soft | boundary | type_diff | miss | novel_candidate | low_confidence |
| One or two agent snapshots | Each: span text, anchor, offsets, concept_name, status |
The disagreement_kind tells you what to focus on:
hard — agents agreed on boundaries + entity_type but disagreed on concept_name. Pick the most-defensible concept under the ontology subtree.soft / boundary — overlapping spans with concept mismatch or boundary jitter. Identify which boundary best matches the entity, and which concept fits.type_diff — same boundaries, different entity_type. Pick the more-specific applicable entity_type.miss — only one side has this span. Decide whether it's a real entity (the other side missed it) or a false positive (the side that emitted it was wrong).novel_candidate — one agent marked it as not-in-the-ontology. Decide whether it really has no ontology match, or whether you can map it to an existing concept the agent missed.low_confidence — single agent emitted the span but flagged uncertainty. Verify against the ontology + note.
Workflow
- Read the patient note at
notes/<note_id>.txt. Locate the span by its
(start, end) offsets and verify the text matches.
- Consult the task's ontology guidance in
references/ontology/ if present, and the scope-skill's SKILL.md
for entity-type-specific rules.
- If the disagreement is on
concept_name, walk the ontology subtree
for the span's entity_type to find the most-specific applicable concept.
- Form your opinion:
- Which agent (if applicable) made the right call?
- What is the single best (entity_type, concept_name, status) tuple
for this span?
- How confident are you in your judgment?
- Emit the strict-JSON record wrapped in
<JUDGE_ANALYSIS>...</JUDGE_ANALYSIS>.
Output schema (strict JSON)
<JUDGE_ANALYSIS>
{
"suggested_concept_name": "Age" | "" (for novel),
"suggested_entity_type": "Demographic",
"suggested_status": "mapped" | "novel_candidate" | "rejected",
"reasoning": "two-to-five sentences explaining the call",
"evidence_pointers": [
{ "note_id": "2024-08-22__ct_chest", "what_to_look_for": "67M on line 4", "offsets": [99, 101] }
],
"agent_correctness": "agent_a" | "agent_b" | "neither" | "both" | "n_a",
"classification_hint": "guideline_gap" | "agent_a_error" | "agent_b_error" | "true_ambiguity" | "novel_concept_candidate" | "n_a",
"judge_confidence": "low" | "medium" | "high"
}
</JUDGE_ANALYSIS>
suggested_concept_name is the canonical label from the ontology subtree
(or empty string if the span is genuinely novel to the ontology).
classification_hint = "novel_concept_candidate" is the NER-specific
classification — it means "this span doesn't fit any existing concept;
the methodologist should consider extending the ontology" and feeds the
chart-review-ner-ontology-extend skill in Phase 2.9.
Hard rules
- Read-only. Never call
set_span_label, set_span_status, or any
write tool. Your only output is the JSON sentinel.
- One record per call. Do not emit multiple records or freeform
commentary outside the sentinels.
- Be specific. Prefer the most-specific ontology concept that
applies; only fall back to the parent when the chart genuinely
underspecifies.
- Cite offsets when you can. Reviewers click
evidence_pointers to
jump to the source location.
