deidentify-a-dataset

# De-identify a dataset Keep the source local, name the free-text columns explicitly, and write to a different destination. Never infer columns or print source and redacted cell values. ## Procedure 1. Confirm that the input is CSV, JSONL/NDJSON, or Parquet. 2. Confirm which columns contain free text. Do not scan or log values to guess. 3. Choose a policy and language. Prefer `strict_no_leak` when recall is the governing safety requirement. 4. Write to a new path; never overwrite the input. 5. Inspect only `result.summary`, which contains aggregate counts and rates. 6. Validate recall and residual leakage on representative synthetic or approved evaluation fixtures before releasing the output. ## Runnable synthetic example Install the model runtime first with `python -m pip install "openmed[hf]"`. ```python import csv from pathlib import Path from openmed import redact_dataset source = Path("synthetic-notes.csv") destination = Path("synthetic-notes.redacted.csv") with source.open("w", newline="", encoding="utf-8") as handle: writer = csv.DictWriter(handle, fieldnames=["record_id", "note"]) writer.writeheader() writer.writerows( [ { "record_id": "SYNTH-001", "note": ( "Taylor Example called 212-555-0198 about a " "metformin refill." ), }, { "record_id": "SYNTH-002", "note": ( "Send the synthetic follow-up to " "demo.patient@example.test." ), }, ] ) result = redact_dataset( source, text_columns=["note"], output_path=destination, policy="strict_no_leak", lang="en", ) print(result.output_path) print(result.summary.to_dict()) # Aggregate counts only; no cell contents. ``` Use the equivalent CLI for an existing dataset: ```bash openmed redact-dataset notes.csv \ --text-columns note,comment \ --policy strict_no_leak \ --output notes.redacted.csv ``` ## Safety checks - Keep model inference and files on infrastructure the user controls. - Do not print input rows, detected entity surfaces, reversible mappings, or exception payloads that may contain source text. - Keep source and output paths separate and access-controlled. - Treat the aggregate summary as evidence, not as proof of compliance. - Never commit real clinical data or restricted evaluation corpora. ## Repository example Read and run [the offline dataset walkthrough](../../examples/datasets_walkthrough.py) when you need a bundled synthetic fixture and first-run download controls.