auditing-deidentification-runs

# Auditing de-identification runs `deidentify(..., audit=True)` returns an `AuditReport` instead of the rewritten text: a deterministic, **PHI-free** record of every redaction decision — offsets, label, detector confidence and threshold, the action taken, content hashes, model provenance, and projected residual risk. Sign it to make it tamper-evident, retain it as compliance evidence, and verify it later without ever touching the original text. Runs **on-device**. ## When to use this skill Use it whenever a de-identification needs to be *defensible*: regulatory retention, internal review, reproducibility checks, or proving to an auditor that identifiers were detected and handled — all without storing or exposing the PHI itself. ## Quick start ```python import openmed note = "Patient John Doe (MRN 1234567) seen 2024-03-02. SSN 123-45-6789." # Returns an AuditReport, NOT a DeidentificationResult, when audit=True. report = openmed.deidentify(note, policy="hipaa_safe_harbor", audit=True) # Make it tamper-evident with a release HMAC key (keep the key in a vault). report.sign(b"my-release-hmac-key", key_id="release-2026") # Persist the no-PHI report (no plaintext identifiers inside). import json with open("deid_audit.json", "w") as fh: json.dump(report.to_dict(), fh, indent=2) # Later: verify integrity. Optionally bind to the exact texts via their hashes. ok = report.verify(b"my-release-hmac-key", original_text=note) assert ok ``` ## What an AuditReport contains `AuditReport` (from `openmed.core.audit`) carries no plaintext PHI. Key fields: | Field | Meaning | | --- | --- | | `policy` | policy profile name in effect | | `resolved_profile` | the concrete settings applied (method, model, thresholds, language, sweep flags) | | `detectors` | `list[DetectorInfo]` — provenance of each detector | | `safety_sweep` | structured-ID sweep metadata (patterns version, source) | | `spans` | `list[AuditSpan]` — one entry per detected identifier | | `thresholds` | per-label confidence thresholds used | | `residual_risk` | projected leakage / re-identification risk summary | | `openmed_version` | library version that produced the report | | `manifest_hash` | hash of the model manifest used | | `document_length` | character length of the input | | `input_hash` | `sha256:` hash of the original text | | `deidentified_text_hash` | `sha256:` hash of the de-identified output | | `repro_hash` | deterministic hash over the canonical payload | | `signature` | `AuditSignature` once `.sign()` is called (else `None`) | `AuditSpan` (per identifier, no plaintext): `start`, `end`, `label`, `canonical_label`, `sources`, `confidence`, `threshold`, `action`, `surrogate`, `text_hash` (hash of the span text), plus `evidence` and a redacted `context`. `DetectorInfo`: `source`, `model_id`, `model_format`, `commit`, `metadata`. `AuditSignature`: `key_id`, `algorithm` (HMAC-SHA256), `value` (hex digest). ## Why audits store offsets + hashes + provenance, never plaintext The whole point of de-identification is to *not* retain identifiers. An audit log full of raw names and MRNs would itself be a PHI breach. So OpenMed records: - **offsets** (`start`/`end`) — *where* an identifier was, not what it was; - **hashes** (`input_hash`, `deidentified_text_hash`, per-span `text_hash`) — prove what was processed without revealing it; - **provenance** (`detectors`, `manifest_hash`, `openmed_version`) — *which* model/version made each call, for reproducibility; - **residual risk** (`residual_risk`) — projected leakage so a reviewer can judge whether the output is safe to release. This lets an auditor confirm coverage and integrity while the report stays shareable. ## Workflow 1. **Run with `audit=True`** and a `policy=` to get an `AuditReport`. 2. **Inspect `residual_risk`** before releasing the output — non-zero projected leakage means review (cross-check with `auditing-safe-harbor-checklist`). 3. **Sign** with `report.sign(key, key_id=...)` using a secret release HMAC key stored in a vault (never in the repo). 4. **Persist** `report.to_dict()` (JSON). For a reviewer view without full context, use `report.export_review_bundle()`. 5. **Verify on retrieval** with `report.verify(key)`. Pass `original_text=` / `deidentified_text=` to also confirm the texts match the recorded hashes. 6. **Retain** per your retention schedule alongside the de-identified output (but keep any reversible mapping separate and encrypted). ## Hand-off to / from OpenMed - **Produce the de-id:** `deidentifying-clinical-text` (`deidentify`); add `audit=True` to get the report from the same call. - **Coverage check:** `auditing-safe-harbor-checklist` maps span labels to the 18 HIPAA identifier categories and flags gaps. - **Policies:** `configuring-privacy-policies` — `resolved_profile` reflects the chosen `policy=`. - **Other surfaces:** MCP `openmed_deidentify` and REST `POST /pii/deidentify` drive the same engine; audit output is available through them. ## Edge cases & gotchas - **`audit=True` changes the return type** to `AuditReport`. You get the report, not `.deidentified_text`. Run a separate normal call if you also need the text. - **An unsigned report is not tamper-evident.** `.sign()` is a deliberate second step; `signature` is `None` until you call it. Empty/None keys are rejected. - **`verify` needs the same key.** It checks the HMAC and the `repro_hash`; optionally it re-hashes texts you pass to confirm they are the audited ones. - **Never put plaintext PHI back into the report.** Do not stuff raw identifiers into `metadata`/`evidence`; the design is hash-and-offset only. - **Store the signing key in a secret manager**, not in source or the report. - **Residual risk is advisory, not a pass/verdict.** Combine it with the Safe Harbor checklist and human review for release decisions. ## Standards & references - HIPAA de-identification & documentation, 45 CFR 164.514(b): https://www.hhs.gov/hipaa/for-professionals/privacy/special-topics/de-identification/index.html - HMAC-SHA256: RFC 2104 / FIPS 198-1. - OpenMed source: `openmed/core/audit.py` (`AuditReport`, `AuditSpan`, `AuditSignature`, `DetectorInfo`, `sign`, `verify`), `openmed/core/pii.py` (`deidentify(audit=True)`).