bio-imaging-mass-cytometry-cell-segmentation

name	bio-imaging-mass-cytometry-cell-segmentation
description	Segment single cells from multiplexed IMC/MIBI tissue images using Mesmer/DeepCell, Cellpose, or ilastik+CellProfiler, covering whole-cell vs nuclear segmentation, the summed-membrane-channel decision, nuclear-expansion bias, lateral spillover, resolution-floor parameters, and downstream-proxy evaluation. Use when delineating cells after preprocessing, choosing a segmentation model, building a cell mask for quantification, diagnosing impossible double-positive populations, or troubleshooting over/under-segmentation.
origin	openai4s
category	bioskills/imaging-mass-cytometry
metadata	{"tool_type":"python","primary_tool":"deepcell","third_party":{"name":"GPTomics/bioSkills","repository":"https://github.com/GPTomics/bioSkills","commit":"d91ed3d563019e649dc854c56ccd62551359488a","license":"MIT"}}

Version Compatibility

Reference examples tested with: steinbock 0.16+, DeepCell 0.12+ (Mesmer), Cellpose 3.0+, numpy 1.26+, scikit-image 0.22+

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Notes specific to this skill: Mesmer expects (batch, y, x, 2) with channel 0 = nuclear, channel 1 = membrane, and image_mpp set to the TRUE acquisition resolution (~1.0 for IMC) -- it was trained at model_mpp ~= 0.5 and rescales the input, so a wrong mpp degrades everything. Cellpose cyto trains at 30-px and nuclei at 17-px diameter; steinbock feeds nuclear-first (reversed vs native Cellpose). Recent steinbock cellpose containers default to the cpsam (Cellpose-SAM) model -- pin the version.

Cell Segmentation for IMC

"Segment cells from my IMC images" -> Draw a per-cell boundary mask so that averaging the channels inside each mask yields single-cell expression.

• Python: deepcell.applications.Mesmer().predict(...), cellpose.models
• CLI: steinbock segment deepcell, steinbock segment cellpose

The Single Most Important Modern Insight -- segmentation is the largest irreversible error source, not a preprocessing step

A single-cell table is literally for each mask_id: mean(pixels_in_mask, every_channel), so the mask defines the support of every measurement and no downstream step -- clustering, batch correction, differential abundance -- can recover a cell the mask merged or split. Two failure modes, and their asymmetry dictates how to tune. Under-segmentation (two cells in one mask) produces LOUD, catchable artifacts: a mask spanning a T cell and a macrophage reports CD3+CD68+, so biologically-impossible co-expression is a segmentation diagnosis until proven otherwise, not a discovery. Over-segmentation (one cell fragmented) is the QUIET, dangerous error: each fragment still looks like a plausible cell, but counts inflate and spatial-neighborhood statistics corrupt without obvious tells. Tuning a watershed or threshold until masks "look clean" usually trades the loud error for the quiet one, which is worse for spatial work. A second, independent problem rides on top: lateral (spatial) spillover -- real signal from a neighbor's membrane bleeding across the shared boundary at ~1 um resolution -- produces the same impossible-co-expression signature even with flawless masks and perfect channel compensation, so the two are confounded and must be addressed separately (REDSEA after segmentation; channel compensation before aggregation).

Methods Landscape

Tool / model	Class	Input it consumes	Strength	Fails when
Mesmer / DeepCell (Greenwald 2022)	deep, trained on TissueNet (incl. IMC/MIBI)	2-ch: nuclear + summed membrane	purpose-built for multiplexed tissue; the IMC default	summed membrane channel is weak/patchy; wrong image_mpp
Cellpose / cpsam (Stringer 2021; Pachitariu 2025)	deep, flow-field / SAM backbone	1-2 ch (cyto +- nuclear)	generalist; cpsam needs no diameter	default models carry a non-IMC size prior; wrong diameter
StarDist (Schmidt 2018)	star-convex polygon regression	single nuclear ch	excellent for crowded round nuclei	nuclear-only; breaks on irregular/elongated cells
ilastik + CellProfiler (Berg 2019; McQuin 2018)	random-forest pixels -> watershed	painted nucleus/cyto/background	transparent, tunable, no GPU; original IMC pipeline	semantic not instance; seed-threshold-sensitive; manual tuning

Decision Tree by Scenario

Scenario	Recommended	Why
Whole-cell phenotyping with a good broadly-expressed membrane marker set	Mesmer whole-cell, image_mpp = true resolution	TissueNet includes this modality; first choice for IMC/MIBI
Membrane staining weak/patchy/cell-type-specific	Nuclei (StarDist/Mesmer-nuclear) + small constrained expansion	a poor membrane sum systematically under-segments types lacking a marker
Only nuclear/intracellular markers needed (TFs, Ki-67)	Nuclear segmentation, quantify directly	nuclear markers barely suffer lateral spillover -- sidesteps the boundary problem
Mesmer struggles on the tissue	Cellpose / cpsam, optionally retrain (Cellpose 2.0)	a panel-specific learned prior beats a wrong generalist prior
Legacy / no-GPU / need full transparency	ilastik -> CellProfiler watershed	the original Bodenmiller pipeline; fully tunable
Impossible co-expression appears after any path	re-tune and/or REDSEA before clustering	the rate is the headline under-segmentation/spillover metric

Whole-Cell Segmentation with Mesmer

Goal: Produce whole-cell instance masks for surface-marker phenotyping.

Approach: Stack nuclear and summed-membrane channels as (batch, y, x, 2) and pass the true acquisition resolution as image_mpp. Mesmer internally rescales to its training resolution, so the mpp is load-bearing, not cosmetic.

import numpy as np
from deepcell.applications import Mesmer

nuclear = img[dna_idx]                       # DNA/Ir channel
membrane = build_membrane(img, membrane_idx) # broadly-expressed membrane sum (see below)
stack = np.stack([nuclear, membrane], axis=-1)[np.newaxis, ...]  # (1, y, x, 2)

app = Mesmer()
masks = app.predict(stack, image_mpp=1.0, compartment='whole-cell')[0, ..., 0]  # ~1.0 for IMC

Build the Summed Membrane Channel

Goal: Construct channel 2 so whole-cell masks are not biased against cell types lacking a marker.

Approach: Sum BROADLY-expressed membrane markers chosen to cover every cell type present (not only the types of interest), because a cell-type-specific sum is bright on some types and dark on others, systematically under-segmenting the dark ones.

def build_membrane(img, membrane_idx):
    # sum pan-membrane markers covering ALL populations (e.g. pan-cytokeratin for
    # epithelium, CD45 for immune, E-cadherin, Na/K-ATPase) -- inspect the result
    # before trusting whole-cell masks; a patchy sum collapses into nuclear-like masks
    return img[membrane_idx].sum(axis=0)

Orchestrate via steinbock

# Mesmer/DeepCell (nuclear-first); membrane channels are aggregated per the panel column
steinbock segment deepcell --minmax -o masks

# Cellpose container (current default model is cpsam; channel order is reversed vs native)
steinbock segment cellpose --minmax -o masks

# aggregate per-cell mean intensities (mean is the default and the right phenotyping choice)
steinbock measure intensities -o intensities

Nuclear Segmentation with Constrained Expansion (fallback)

Goal: Approximate whole cells when membrane staining is absent, without the bias of free dilation.

Approach: Segment nuclei, then expand with a small radius under a competitive/watershed constraint so pixels are owned by exactly one cell. Fixed isotropic dilation is a cell-type-correlated bias (under-captures macrophages, over-captures small cells) and free dilation double-counts boundary pixels into two masks.

from skimage.segmentation import expand_labels, watershed

# expand_labels grows each label into background but stops at the midline between
# labels (no overlap), so each pixel is assigned once -- a partition, unlike free dilation
expanded = expand_labels(nuclear_masks, distance=3)   # ~3 px at 1 um; report the radius
assert expanded.max() == nuclear_masks.max()          # no cells created/destroyed

Per-Tool Failure Modes

Mesmer -- wrong image_mpp

Trigger: leaving the default mpp on 1 um IMC. Mechanism: Mesmer rescales the image to its ~0.5 um training resolution; a wrong mpp rescales cells to the wrong learned size. Symptom: systematic over/under-segmentation across the whole image. Fix: pass image_mpp = true acquisition resolution (~1.0 IMC, ~0.5 or finer MIBI).

Cellpose -- auto-diameter on tiny cells

Trigger: auto-diameter on ~5-px IMC nuclei. Mechanism: cyto rescales to a 30-px target; at single-digit diameters the rescale factor is large and unstable. Symptom: merged or fragmented masks. Fix: set diameter from known cell size in pixels, or use cpsam (no diameter dependence).

Generalist model -- wrong size prior

Trigger: default Cellpose cyto/cyto3 on IMC, trusted blindly. Mechanism: at ~5 px of evidence the learned PRIOR, not the image, draws the boundary, and a non-IMC prior is wrong. Symptom: plausible-looking but systematically biased masks. Fix: prefer Mesmer (TissueNet includes IMC/MIBI) or fine-tune Cellpose on the panel; evaluate on downstream proxies.

Channel compensation in the wrong order

Trigger: REDSEA before segmentation, or channel compensation after aggregation. Mechanism: channel spillover is pixel-level (must be corrected before the per-cell average); lateral spillover is defined on segmented neighbors (must be corrected after). Symptom: residual impossible co-expression. Fix: pixel-compensate -> segment -> aggregate -> REDSEA.

Quantitative Thresholds

Threshold	Source	Rationale
image_mpp ~= 1.0 (IMC)	Greenwald 2022; Mesmer model_mpp ~0.5	match acquisition resolution to the rescaler
Cellpose cyto 30 px / nuclei 17 px	Stringer 2021	the trained-diameter targets the model rescales to
Nuclear expansion ~3 px @ 1 um	ImcSegmentationPipeline convention	approximates a thin cytoplasm without crossing into neighbors
Lymphocyte ~5-7 px @ 1 um/px	Giesen 2014	the resolution floor that makes size priors load-bearing
Impossible-co-expression rate	Bai 2021	per-slide under-segmentation/lateral-spillover monitor; not an F1 substitute

Common Errors

Error / symptom	Cause	Solution
CD3+CD68+ "hybrid" cluster	under-segmentation or lateral spillover	treat as QC failure; re-tune + REDSEA before clustering
Whole-cell masks collapse to nuclei	weak/patchy summed membrane channel	broaden the membrane sum or fall back to nuclei + expansion
Same pixel counted in two cells	free dilation expansion	use expand_labels/watershed (exclusive ownership); assert label count unchanged
Macrophages under-captured	fixed isotropic nuclear dilation	constrained expansion; accept and report the bias; don't cross-compare with whole-cell data
Native Cellpose channel args do nothing in steinbock	steinbock reverses channel order	configure channels via the steinbock panel column, not native --chan semantics
High IoU but wrong biology	optimizing a pixel-overlap metric	accept on downstream proxies (impossible-co-expression rate, count/density sanity, positive-fraction stability); audit dense regions

References

• Giesen C, Wang HAO, Schapiro D, et al. 2014. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat Methods 11(4):417-422. — IMC ~1 um resolution floor.
• Berg S, Kutra D, Kroeger T, et al. 2019. ilastik: interactive machine learning for (bio)image analysis. Nat Methods 16(12):1226-1232. — pixel classification stage.
• McQuin C, Goodman A, Chernyshev V, et al. 2018. CellProfiler 3.0: Next-generation image processing for biology. PLoS Biol 16(7):e2005970. — watershed instance segmentation.
• Schmidt U, Weigert M, Broaddus C, Myers G. 2018. Cell Detection with Star-Convex Polygons. MICCAI 2018, LNCS 11071:265-273. — StarDist nuclear baseline.
• Stringer C, Wang T, Michaelos M, Pachitariu M. 2021. Cellpose: a generalist algorithm for cellular segmentation. Nat Methods 18(1):100-106. — Cellpose diameters.
• Pachitariu M, Rariden M, Stringer C. 2025. Cellpose-SAM: superhuman generalization for cellular segmentation. bioRxiv doi:10.1101/2025.04.28.651001. — the cpsam SAM-backbone model (preprint).
• Greenwald NF, Miller G, Moen E, et al. 2022. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat Biotechnol 40(4):555-565. — Mesmer/DeepCell, TissueNet, image_mpp.
• Bai Y, Zhu B, Rovira-Clave X, et al. 2021. Adjacent Cell Marker Lateral Spillover Compensation and Reinforcement for Multiplexed Images. Front Immunol 12:652631. — REDSEA boundary compensation; lateral spillover signature.
• Windhager J, Zanotelli VRT, Schulz D, et al. 2023. An end-to-end workflow for multiplexed image processing and analysis. Nat Protoc 18(11):3565-3613. — steinbock segmentation/measurement.

Related Skills

• data-preprocessing - channel spillover compensation precedes segmentation
• phenotyping - consumes the single-cell mask and intensities; double-positives diagnose segmentation
• spatial-analysis - over-segmentation corrupts neighborhood statistics
• quality-metrics - segmentation QC metrics and the impossible-co-expression monitor
• interactive-annotation - overlay masks on channels to audit boundaries