| name | building-gold-corpus |
| description | Scaffold a synthetic gold-standard annotation project for evaluating OpenMed NER and de-identification models — label schema, annotation guidelines, BRAT or Label Studio config, and disjoint train/dev/test splits. Use when the user wants to create eval fixtures, set up annotation, define a label set, write guidelines, configure an annotation tool, or build a held-out gold set for the OpenMed eval harness. Trigger on "gold corpus", "annotation project", "label schema", "annotation guidelines", "BRAT", "Label Studio", "train dev test split", or "build eval fixtures" for OpenMed. Committed gold must be synthetic; licensed (i2b2/n2c2/MIMIC) data is eval-only and never committed. |
| license | Apache-2.0 |
| metadata | {"project":"OpenMed","category":"evaluation-quality","pairs":"adjacent","version":"1.0"} |
Building a Gold Corpus
You can't evaluate what you can't measure against. This skill scaffolds a
gold-standard annotation project whose output drops straight into the OpenMed
eval harness as fixtures. The hard rule: anything committed to the repo is
synthetic. Licensed clinical corpora (i2b2, n2c2, MIMIC) are DUA-gated — use
them at eval time from the user's own copy, never check them in.
When to use this skill
- You need eval fixtures for
benchmarking-clinical-ner or
evaluating-with-leakage-gates and have none.
- You're standing up an annotation effort: schema, guidelines, tool config.
- You need disciplined train/dev/test splits with no leakage between them.
- You want a small synthetic golden set you can commit and gate on in CI.
The OpenMed fixture shape (your target output)
Annotations must serialize to character-offset spans the harness understands:
{
"fixtures": [
{
"id": "synthetic-0001",
"language": "en",
"text": "Ms. Jane Roe (MRN 0000000) seen 2099-01-02 for type 2 diabetes.",
"gold_spans": [
{"start": 4, "end": 12, "label": "PERSON"},
{"start": 18, "end": 25, "label": "ID_NUM"},
{"start": 32, "end": 42, "label": "DATE"},
{"start": 47, "end": 62, "label": "DISEASE"}
]
}
]
}
openmed.eval.harness.load_fixtures accepts a top-level list or a {"fixtures": [...]} mapping. Offsets are character indices into text; labels are
OpenMed-canonical.
Quick start — scaffold the project
eval/
gold/
guidelines.md # annotation manual + edge-case decisions
label_schema.json # canonical labels + definitions + examples
synthetic/ # COMMITTED synthetic fixtures (CI-gateable)
train.json
dev.json
test.json
external/ # GITIGNORED: licensed DUA corpora, eval-only
.gitignore # * (never commit i2b2/n2c2/MIMIC)
Verify your synthetic fixtures load and validate spans before you trust them:
from openmed.eval.harness import load_fixtures
fixtures = load_fixtures("eval/gold/synthetic/test.json")
print(len(fixtures), "fixtures;", sum(len(f.gold_spans) for f in fixtures), "spans")
Workflow
- Define the label schema. Reuse OpenMed canonical labels (PERSON, DATE,
ID_NUM, EMAIL, PHONE, DISEASE, DRUG, ...). Each label gets a one-line
definition, in/out examples, and a boundary rule (include titles? trailing
punctuation?).
- Write annotation guidelines. The manual is the contract: span boundaries,
nested/overlapping policy, ambiguous cases, and a decision log appended as
real cases force calls. Vague guidelines → low agreement → unusable gold.
- Generate synthetic source text. Compose realistic clinical narratives
with fabricated identifiers (Faker-style names, impossible dates like
2099-, all-zero MRNs). Never paste real notes into committed data.
- Configure the tool. BRAT uses
annotation.conf (entity types) producing
.ann standoff; Label Studio uses a labeling-config XML producing JSON.
Map either back to the fixture shape above.
- Double-annotate and measure agreement. Have ≥2 annotators on an overlap
set; compute span-level inter-annotator agreement (F1 or Cohen's κ). Adjudicate
disagreements and fold the resolutions into the decision log.
- Split with discipline. Partition by document/patient, not by sentence,
so no patient appears in two splits. Freeze test; never tune on it.
- Validate and commit. Run
load_fixtures; confirm spans align and ids are
unique. Commit only the synthetic splits.
Hand-off to / from OpenMed
- To
benchmarking-clinical-ner: dev/test fixtures feed run_suite and
error_report for the NER scorecard.
- To
evaluating-with-leakage-gates and gating-deid-leakage: the synthetic
held-out set is exactly what the release gates and the CI gate run against.
- To
building-with-openmed: synthetic notes can be generated by running
surrogate replacement through openmed.deidentify(method="replace").
- Pairs with
auditing-subgroup-fairness: tag each gold span with a group
in metadata so fairness_report can slice by demographic surrogate.
Edge cases & gotchas
- Committed = synthetic. No exceptions. Real PHI in the repo is a breach even
if the repo is private. Generate identifiers; don't transcribe them.
- DUA data is eval-only. Load i2b2/n2c2/MIMIC from
eval/external/ (gitignored)
at runtime under the user's license; results may be reported, data never shared.
- Split by patient, not by line. Sentence-level splitting leaks a patient's
style/identifiers across train and test and inflates scores.
- Offsets must be character indices into this
text. Re-tokenization or
whitespace edits silently shift offsets; re-validate with load_fixtures.
- Label the fairness surrogate, not real demographics. Put a synthetic
group tag in span metadata; don't store real protected attributes.
- Decision log is the gold's source of truth. Without it, two re-annotations
disagree and your "ceiling" F1 is noise.
Standards & references
