running-openmed-ondevice

# Running OpenMed on-device OpenMed runs **fully on-device** by design. These three backends let you take it further at the edge: **MLX** (Apple Silicon acceleration), **CoreML** (iOS/macOS / Neural Engine), and **ONNX / WebGPU** (cross-platform and in-browser). The flow is the same: **convert → (quantize) → run locally**. Because inference is local, raw PHI never leaves the device — the strongest privacy posture OpenMed offers. ## When to use this skill When you need OpenMed where there is no server: an iOS/macOS app (CoreML), fast NER/de-id on an Apple Silicon Mac (MLX), or a portable/browser deployment (ONNX/WebGPU). For a hosted endpoint use `serving-openmed-rest-api`; for an agent tool use `deploying-openmed-mcp`; for corpora use `batch-processing-clinical-text`. ## Pick a backend | Backend | Extra | Best for | Quantization | | --- | --- | --- | --- | | **MLX** | `openmed[mlx]` | Apple Silicon Macs; fastest local NER/de-id; on-device LLMs | 4-bit / 8-bit weights | | **CoreML** | `openmed[coreml]` | iOS/iPadOS/macOS apps, Neural Engine | int8 palettization | | **ONNX / WebGPU** | `openmed[onnx]` | cross-platform runtimes, browser (transformers.js) | fp16 (WebGPU); int8 via ORT | ## Quick start — MLX (Apple Silicon) ```bash pip install "openmed[mlx]" # Convert a HF token-classification model to an OpenMed MLX artifact, 8-bit: python -m openmed.mlx.convert --model OpenMed/<some-ner-model> --output ./mlx_ner --quantize 8 ``` ```python import openmed # Run NER/de-id through the normal API — pass the local artifact dir as model_name. # The loader auto-detects the MLX backend from the artifact (or set backend explicitly). result = openmed.analyze_text( "Patient received 75mg clopidogrel for NSTEMI.", model_name="./mlx_ner", # local MLX artifact directory output_format="dict", ) # Force MLX via config if you prefer to be explicit: from openmed.core.config import OpenMedConfig cfg = OpenMedConfig(backend="mlx") # None=auto-detect, "mlx", or "hf" ``` `convert()` is also importable: `openmed.mlx.convert.convert(model_id, output_dir, quantize_bits=8)`. The CLI accepts `--quantize {4,8}`, `--quantize-group-size`, `--cache-dir`, and an optional `--eval-suite` to **certify quantized recall** against the full-precision parent (recommended for clinical models — quantization can drop recall on rare entities). ### On-device LLM generation (MLX) ```python from openmed.mlx.lm import generate_text, OpenMedMLXLanguageModel text = generate_text( messages=[{"role": "user", "content": "Summarize: chest pain, troponin elevated."}], model_name="OpenMed/laneformer-2b-it-q4-mlx", # resolves to a local MLX-LM artifact max_tokens=128, ) llm = OpenMedMLXLanguageModel("OpenMed/laneformer-2b-it-q4-mlx") out = llm.generate(prompt="...", max_tokens=64, temp=0.0) ``` ## Quick start — CoreML (iOS/macOS) ```bash pip install "openmed[coreml]" python -m openmed.coreml.convert --model OpenMed/<some-ner-model> --output model.mlpackage --quantize int8 ``` ```python from openmed.coreml.convert import convert convert( "OpenMed/<some-ner-model>", "model.mlpackage", compute_units="cpuAndNeuralEngine", # "all" | "cpuAndNeuralEngine" | "cpuOnly" compute_precision="float16", # float16 for Neural Engine, float32 for CPU quantize="int8", # emits an int8-palettized sibling .mlpackage ) ``` Bundle the `.mlpackage` in your Xcode app and run it with Core ML; the converter writes the `id2label` map so your app can decode token labels. Use `float16` + `cpuAndNeuralEngine` for the Neural Engine; `int8` shrinks the model for storage-constrained devices. ## Quick start — ONNX / WebGPU ```bash pip install "openmed[onnx]" python -m openmed.onnx.convert --model OpenMed/<some-ner-model> --output ./onnx_out ``` ```python from openmed.onnx.convert import convert res = convert("OpenMed/<some-ner-model>", "./onnx_out", include_webgpu=True, opset=18) # Emits model.onnx (fp32) and model.webgpu.onnx (fp16) + an export manifest. ``` Run `model.onnx` with ONNX Runtime on any platform, or ship `model.webgpu.onnx` to the browser via transformers.js for in-page, zero-upload inference. Use `--no-webgpu` to skip the fp16 artifact. ## Workflow 1. **Pick the backend** for the target (table above). 2. **Convert** the HF/OpenMed model with the matching `convert()` / `python -m openmed.<backend>.convert`. 3. **Quantize** if size/latency demands it (MLX 4/8-bit, CoreML int8, WebGPU fp16). For clinical de-id/NER, **certify recall** — MLX's `--eval-suite` writes a recall-delta report so you don't silently lose rare entities. 4. **Run locally:** MLX artifacts go straight through `analyze_text` / `deidentify`; CoreML/ONNX artifacts run in their native runtimes (Core ML, ONNX Runtime, transformers.js). 5. **Verify** outputs against the full-precision model before shipping (`evaluating-with-leakage-gates` for de-id). ## Hand-off to / from OpenMed - **Same API surface:** an MLX artifact path is a drop-in `model_name` for `openmed.analyze_text` / `deidentify` — downstream skills (`building-patient-timelines`, `exporting-to-fhir`) are unchanged. - **From the catalog:** start from a model chosen via `choosing-openmed-models` / `loading-openmed-models`, then convert it here. - **Eval gate:** pipe quantized de-id output into `evaluating-with-leakage-gates` before release. ## Edge cases & gotchas - **Quantization can hurt clinical recall.** A dropped rare PHI entity is a breach. Always benchmark the quantized model vs. full precision (MLX `--eval-suite`/recall-delta; manual eval for CoreML/ONNX) and gate on leakage, not just F1. - **MLX is Apple-Silicon only.** On non-Apple hardware the MLX backend isn't available and OpenMed falls back to PyTorch; convert/quantize steps that need `mlx` will skip quantization with a warning. - **CoreML compute units matter.** `float16` targets the Neural Engine but some ops fall back to CPU; validate latency on a real device, not just the simulator. - **ONNX dynamic axes / opset.** Keep `opset>=18` and verify the model with `onnx.checker` (the converter does). token-classification only — these converters wrap `AutoModelForTokenClassification`. - **On-device ≠ no responsibility.** Local inference removes network exposure, but the model and any cached output still live on the device — encrypt at rest and keep raw PHI out of logs. - **No license bundling.** Convert your own permissively-licensed models; don't embed restricted terminologies in shipped artifacts. ## Standards & references - Apple MLX: https://github.com/ml-explore/mlx · MLX-LM: https://github.com/ml-explore/mlx-lm - Core ML Tools: https://apple.github.io/coremltools/ - ONNX: https://onnx.ai/ · ONNX Runtime: https://onnxruntime.ai/ - WebGPU in the browser via transformers.js: https://huggingface.co/docs/transformers.js - OpenMed source: `openmed/mlx/convert.py` & `openmed/mlx/lm.py` (`convert`, `generate_text`, `OpenMedMLXLanguageModel`), `openmed/coreml/convert.py` (`convert`), `openmed/onnx/convert.py` (`convert`, `export_onnx`, `export_webgpu`), `openmed/core/backends.py` (auto-detect), `openmed/core/config.py` (`backend`).