deidentifying-clinical-text

# De-identifying clinical text `openmed.deidentify` detects PHI/PII and rewrites the text so it can be shared, stored, or analyzed without exposing patients. It runs **fully on-device** after a one-time model download — no network calls, no telemetry, no raw PHI leaving the process. This is the single most important OpenMed entry point for privacy work; everything else (policies, audit, multilingual, date-shifting) layers on top of it. ## When to use this skill Reach for `deidentify` when you need to transform text — replace, mask, remove, hash, or date-shift the identifiers. If you only need to **locate** PHI spans without changing the text, use `extract_pii` (see `extracting-pii-entities`). To **restore** masked text later, use `reidentify` (see `reidentifying-text`). ## Quick start ```python import openmed note = ( "Patient John Doe (MRN 1234567) was seen on 2024-03-02 by Dr. Alice Reed. " "Contact: john.doe@example.com, 617-555-0142." ) result = openmed.deidentify( note, method="mask", # mask | remove | replace | hash | shift_dates confidence_threshold=0.7, # safety default; raise to reduce false negatives' impact policy="hipaa_safe_harbor", # optional bundled profile (see below) ) print(result.deidentified_text) # Patient [NAME] (MRN [ID_NUM]) was seen on [DATE] by Dr. [NAME]. ... for e in result.pii_entities: # NEVER log e.text / e.original_text — those are raw PHI. Use offsets + label. print(e.canonical_label, e.start, e.end, round(e.confidence, 3)) ``` `deidentify` returns a `DeidentificationResult` with these fields (note the exact names): | Field | What it holds | | --- | --- | | `.deidentified_text` | the rewritten, PHI-safe string (your output) | | `.pii_entities` | `list[PIIEntity]` — each has `start`, `end`, `canonical_label`, `confidence`, `action`, `surrogate`; `original_text`/`text` hold raw PHI | | `.mapping` | redacted→original dict, only when `keep_mapping=True` (secret) | | `.method` | the method actually applied | | `.metadata` | run metadata (model, policy, counts) | ## The five methods | `method=` | Effect | Reversible? | Use when | | --- | --- | --- | --- | | `"mask"` | `John Doe` → `[NAME]` | with `keep_mapping=True` | default; clear that redaction happened | | `"remove"` | deletes the span entirely | no | minimal-footprint output | | `"replace"` | type-matched fake value (`John Doe`→`Mark Lee`) | with `keep_mapping=True` | keep notes readable/parseable (see `generating-synthetic-surrogates`) | | `"hash"` | stable hash per value, links repeats | no (one-way) | cohort linkage without revealing identity | | `"shift_dates"` | moves dates, preserves intervals | n/a | research needing temporal structure (see `shifting-clinical-dates`) | ## Workflow 1. **Pick a method and a policy.** Start from a bundled `policy=` profile (`hipaa_safe_harbor`, `gdpr_pseudonymization`, `research_limited_dataset`, …) so per-label actions are set for you. See `configuring-privacy-policies`. 2. **Set `confidence_threshold` deliberately.** Default is `0.7`. For de-id, prefer *over*-redaction: a missed identifier is a breach, an over-redacted token is just noise. The bundled safety sweep catches structured IDs (SSN, MRN-like, emails) even below threshold. 3. **Run `deidentify`.** Inspect `result.pii_entities` by **offset and label**, not raw text, to confirm coverage. 4. **For stable surrogates**, pass `consistent=True, seed=<int>` so the same input maps to the same fake value every run (reproducible pipelines). 5. **For reversibility**, pass `keep_mapping=True` and store `result.mapping` in a secured vault — never alongside the de-identified output. 6. **Verify, don't assume.** Check residual risk with `audit=True` (`auditing-deidentification-runs`) and the 18-identifier checklist (`auditing-safe-harbor-checklist`). ## Consistent surrogates and reversibility ```python # Same fake identity for every mention of the same person, reproducibly: r = openmed.deidentify(note, method="replace", consistent=True, seed=42) # Reversible de-id (keep the mapping secret and separate from output): r = openmed.deidentify(note, method="mask", keep_mapping=True) restored = openmed.reidentify(r.deidentified_text, r.mapping) assert restored == note ``` ## Hand-off to / from OpenMed - **Detect only:** `openmed.extract_pii(text)` → `PredictionResult` with `.entities` (spans, no rewrite). Use it to preview coverage first. - **Restore:** `openmed.reidentify(deidentified_text, mapping)` — requires `keep_mapping=True` at de-id time and proper authorization. - **Policies:** `configuring-privacy-policies` to choose/customize a `policy=`. - **Audit:** `deidentify(..., audit=True)` → `AuditReport` with offsets, hashes, detector provenance, and residual-risk — never plaintext. - **Other surfaces (same engine):** MCP tool `openmed_deidentify`; REST `POST /pii/deidentify`. There is **no** CLI de-id command. ## Edge cases & gotchas - **Attribute names.** It is `result.deidentified_text` and `result.pii_entities` — not `.text`/`.entities`. (`extract_pii` returns a `PredictionResult` whose spans are at `.entities`.) - **Raw PHI never leaves the span objects.** `PIIEntity.text` and `.original_text` contain real identifiers. Do not print, log, or cache them. Audit and logs use offsets, `canonical_label`, and hashes only. - **Threshold is a safety dial, not an accuracy dial.** Lowering it redacts *more*; in de-id, false positives are cheap and false negatives are breaches. - **`shift_dates` is for dates only**; combine with `keep_year`/`date_shift_days` (see `shifting-clinical-dates`). It does not touch names or IDs. - **`keep_mapping` output is sensitive as PHI.** The mapping re-identifies everyone — store it encrypted, access-controlled, and apart from the output. - **Multilingual:** pass `lang=` (and `locale=` for surrogates) for non-English notes; see `deidentifying-multilingual-text`. Do not run English models on other languages. - **De-id is verified, not assumed.** Gate releases on leakage/residual-risk, not F1 alone. ## Standards & references - HIPAA De-identification, 45 CFR 164.514(b) — Safe Harbor & Expert Determination: https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html - HHS Safe Harbor 18-identifier guidance: see `auditing-safe-harbor-checklist`. - OpenMed source: `openmed/core/pii.py` (`deidentify`, `extract_pii`, `reidentify`, `DeidentificationResult`, `PIIEntity`).