phi-deidentifier

# PHI De-identifier De-identify clinical / health text to the **HIPAA Safe Harbor** standard (or a **Limited Data Set**) using the bundled, offline `scripts/deidentify.py`. Every run produces two things: the **redacted text** and an **audit manifest** listing what was removed, by category, plus anything that needs human review. ## When to use - "De-identify / anonymise / scrub / redact this note (or file / folder)." - "Remove patient identifiers before I share this." - "Make this Safe Harbor compliant" or "produce a limited data set." - Preparing PHI-free samples for demos, test fixtures, analytics, or model prompts. Do **not** use it to answer clinical questions or to claim legal certification — it is a mechanical redactor plus an audit trail for a human to verify. ## How to run it Always call the bundled script; never improvise redaction logic. ```bash # Safe Harbor (default): writes NOTE.deid.txt + NOTE.manifest.json next to input python scripts/deidentify.py NOTE.txt # Limited Data Set: retains dates and city/state/ZIP, strips direct identifiers python scripts/deidentify.py NOTE.txt --mode limited # Inline text, machine-readable result to stdout python scripts/deidentify.py --text "John Doe, SSN 123-45-6789" --json # HYBRID name recall: pass the person names YOU (the agent) spotted in free prose python scripts/deidentify.py --text "<text>" --names "Rafael Alcaraz,Ngozi Okafor" --json # STRUCTURED / JSON input: redact by FIELD NAME (MRN, City, Employer, DOB, ...) # Use this whenever the input is a JSON object or a key/value record — NOT free text. python scripts/deidentify.py intake_record.json --format record --json # ALSO emit a reversible token->value crosswalk (SENSITIVE — re-identifying) python scripts/deidentify.py NOTE.txt --map crosswalk.json ``` Steps: 1. Identify the input (a file path, a folder, or inline text) and the target standard: **Safe Harbor** (default) or **Limited Data Set** (`--mode limited`). 2. **Pick the right mode for the shape of the input.** If the input is a **structured record** — a JSON object, or clear `Field: value` key/value pairs (e.g. an intake record, an API payload, an exported row) — run it with `--format record`. Record mode maps each field by its NAME (MRN, City, Employer, DOB, SSN, …) and redacts deterministically; free-text mode will MISS quoted/structured identifiers like `"MRN": "33108876"` because there is no adjacent label to anchor on. Use the default (free-text) mode only for prose. Still run the name-recall pass on any free-text field VALUES (e.g. a `ChiefComplaint` or `Notes` field that contains a person's name). 3. **Name-recall pass (do this every time on free prose).** Before running, read the text yourself and list every PERSON name and PLACE (city / facility) you can identify — including unusual or non-English names that a regex or the bundled lexicon would miss (e.g. "Rafael Alcaraz", "Okafor"). This is the model/agent contribution to the hybrid; you are the recall booster. 4. Run `scripts/deidentify.py`, passing the names from step 3 via `--names "Name One,Name Two"` (or a `--names-file` with `{"names":[...], "places":[...],"orgs":[...]}`). The script deterministically redacts every occurrence, tokenizes consistently, runs its guard, and writes the manifest. The script — not your paraphrase — is the auditable system of record. 5. Return the **redacted text** exactly as produced, and summarise the **manifest**: how many identifiers of each category were removed. 6. **Surface `needs_human_review`.** Names caught heuristically, and any unknown capitalised text flagged for review, are listed there — tell the user these must be verified before release, and that bare names in free prose may still be missed. 7. Only write the `--map` crosswalk when the user explicitly needs to re-identify later; warn that it is sensitive and must be stored securely, separately from the de-identified output. ## Use as a safety net before an outbound action Agents often carry actions that emit content — **Send Email, Create Record, Post Message, Export**. When an agent maker points one of those actions at this skill (e.g. an instruction like *"before Send Email, de-identify the content using phi-deidentifier"*), apply this skill as a gate: 1. Take the exact text about to leave the agent (e.g. the email Subject + Body, or the field values of the record). 2. De-identify it with the steps above (including the name-recall pass). 3. Populate the action's inputs from the **redacted** result only — never from the original text. 4. If `needs_human_review` is non-empty, **do not fire the action**; show the redacted draft plus the review items and ask the user to confirm first. 5. On success, tell the user the content was de-identified and how many identifiers were removed. Never place the `--map` crosswalk in the outbound content or in chat. The maker only needs to reference this skill by name in the action's instruction; the operational steps above live here, in the skill, so they do not have to be re-written per agent. ## What it removes (HIPAA Safe Harbor, 18 identifiers) Handled **deterministically by regex** (high confidence): names in labelled fields and after titles, street addresses, city + state, geographic ZIP (truncated to 3 digits; restricted prefixes zeroed), all date elements finer than year, ages 90+ (aggregated), phone (US, international `+country-code`, and labelled formats), fax, email, URL, IP address, SSN (dashed, spaced, or labelled — including run-together digits), MRN, health-plan / account / member numbers, certificate / license / DEA / NPI numbers, device serial numbers, VIN. As a backstop, any **unlabelled digit run of 7+ characters** that looks like an identifier the structured detectors couldn't anchor on (a bare account / record / SSN / device or phone number) is **flagged under `needs_human_review`** — never silently dropped, and never auto-redacted (a bare number may be a legitimate non-PHI value, so a human confirms). Handled **best-effort** and flagged for review: person names in free prose. Names are caught by (a) title/label patterns ("Patient: …", "Dr. …") and (b) a bundled offline **name lexicon** (`assets/name_lexicon.txt`) that catches many bare names with no title or label — all fully local, no network or extra packages, so it works in locked-down agent sandboxes (e.g. the Copilot Studio Python container where you cannot `pip install`). Names not in the lexicon can still be missed and word-like names may be over-redacted, so **human review of free text is required**; extend the lexicon for your population. (If a runtime already ships spaCy it is used automatically as a bonus, but installation is never assumed.) City/state written without a comma or trailing ZIP may also be missed. **Full-face photos and biometric identifiers are out of scope** — this skill processes text only; call that out if the source contains images. ## Transformations (so downstream data stays useful) - **Consistent pseudonyms:** each distinct value maps to a stable token, e.g. every occurrence of the same MRN becomes `[MRN-1]`, so linkage is preserved without exposing the value. - **Dates → year only:** `03/15/1985` → `[DATE:1985]` (Safe Harbor keeps year). - **Ages:** ≤89 retained; 90+ → `[AGE:90+]`. - **ZIP → first 3 digits:** `98104` → `[ZIP:981XX]`; HHS-restricted low-population prefixes → `[ZIP:000]`. - **Limited Data Set mode** keeps dates and city/state/ZIP intact while still removing names, contacts, SSN, MRN, and account numbers. ## The audit manifest `NOTE.manifest.json` records the mode, a per-category count summary, one entry per token with a **salted hash** of the original value (never the plaintext), the `needs_human_review` list, and explanatory notes. Use it as the evidence trail for a reviewer — it answers "what did the tool remove, and what still needs eyes?" without itself containing PHI. ## Guardrails - **Never claim compliance.** Report what was removed; a qualified human makes the Safe Harbor / expert-determination call. - **Never invent or "restore" content.** Only redact. - **Do not paste the crosswalk into chat** or co-locate it with the output; it re-identifies the data. - **Always relay `needs_human_review`** and remind the user to review free text — regex cannot guarantee every bare name or unusual identifier is caught. - Text only: if the source has images/scans, say so and recommend a separate image-redaction step. ## References - `references/hipaa-safe-harbor.md` — the 18 identifiers, ZIP/date/age rules, and Safe Harbor vs Limited Data Set, as implemented here.