Bio Imaging Mass Cytometry Cell Segmentation

Version Compatibility

Reference examples tested with: steinbock 0.16+, DeepCell 0.12+ (Mesmer), Cellpose 3.0+, numpy 1.26+, scikit-image 0.22+

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• CLI: <tool> --version then <tool> --help to confirm flags

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Notes specific to this skill: Mesmer expects (batch, y, x, 2) with channel 0 = nuclear, channel 1 = membrane, and image_mpp set to the TRUE acquisition resolution (~1.0 for IMC) -- it was trained at model_mpp ~= 0.5 and rescales the input, so a wrong mpp degrades everything. Cellpose cyto trains at 30-px and nuclei at 17-px diameter; steinbock feeds nuclear-first (reversed vs native Cellpose). Recent steinbock cellpose containers default to the cpsam (Cellpose-SAM) model -- pin the version.

Cell Segmentation for IMC

"Segment cells from my IMC images" -> Draw a per-cell boundary mask so that averaging the channels inside each mask yields single-cell expression.

• Python: deepcell.applications.Mesmer().predict(...), cellpose.models
• CLI: steinbock segment deepcell, steinbock segment cellpose

The Single Most Important Modern Insight -- segmentation is the largest irreversible error source, not a preprocessing step

A single-cell table is literally for each mask_id: mean(pixels_in_mask, every_channel), so the mask defines the support of every measurement and no downstream step -- clustering, batch correction, differential abundance -- can recover a cell the mask merged or split. Two failure modes, and their asymmetry dictates how to tune. Under-segmentation (two cells in one mask) produces LOUD, catchable artifacts: a mask spanning a T cell and a macrophage reports CD3+CD68+, so biologically-impossible co-expression is a segmentation diagnosis until proven otherwise, not a discovery. Over-segmentation (one cell fragmented) is the QUIET, dangerous error: each fragment still looks like a plausible cell, but counts inflate and spatial-neighborhood statistics corrupt without obvious tells. Tuning a watershed or threshold until masks "look clean" usually trades the loud error for the quiet one, which is worse for spatial work. A second, independent problem rides on top: lateral (spatial) spillover -- real signal from a neighbor's membrane bleeding across the shared boundary at ~1 um resolution -- produces the same impossible-co-expression signature even with flawless masks and perfect channel compensation, so the two are confounded and must be addressed separately (REDSEA after segmentation; channel compensation before aggregation).

Methods Landscape

Decision Tree by Scenario

Whole-Cell Segmentation with Mesmer

Goal: Produce whole-cell instance masks for surface-marker phenotyping.

Approach: Stack nuclear and summed-membrane channels as (batch, y, x, 2) and pass the true acquisition resolution as image_mpp. Mesmer internally rescales to its training resolution, so the mpp is load-bearing, not cosmetic.

import numpy as np
from deepcell.applications import Mesmer

nuclear = img[dna_idx]                       # DNA/Ir channel
membrane = build_membrane(img, membrane_idx) # broadly-expressed membrane sum (see below)
stack = np.stack([nuclear, membrane], axis=-1)[np.newaxis, ...]  # (1, y, x, 2)

app = Mesmer()
masks = app.predict(stack, image_mpp=1.0, compartment='whole-cell')[0, ..., 0]  # ~1.0 for IMC

Build the Summed Membrane Channel

Goal: Construct channel 2 so whole-cell masks are not biased against cell types lacking a marker.

Approach: Sum BROADLY-expressed membrane markers chosen to cover every cell type present (not only the types of interest), because a cell-type-specific sum is bright on some types and dark on others, systematically under-segmenting the dark ones.

def build_membrane(img, membrane_idx):
    # sum pan-membrane markers covering ALL populations (e.g. pan-cytokeratin for
    # epithelium, CD45 for immune, E-cadherin, Na/K-ATPase) -- inspect the result
    # before trusting whole-cell masks; a patchy sum collapses into nuclear-like masks
    return img[membrane_idx].sum(axis=0)

Orchestrate via steinbock

# Mesmer/DeepCell (nuclear-first); membrane channels are aggregated per the panel column
steinbock segment deepcell --minmax -o masks

# Cellpose container (current default model is cpsam; channel order is reversed vs native)
steinbock segment cellpose --minmax -o masks

# aggregate per-cell mean intensities (mean is the default and the right phenotyping choice)
steinbock measure intensities -o intensities

Nuclear Segmentation with Constrained Expansion (fallback)

Goal: Approximate whole cells when membrane staining is absent, without the bias of free dilation.

Approach: Segment nuclei, then expand with a small radius under a competitive/watershed constraint so pixels are owned by exactly one cell. Fixed isotropic dilation is a cell-type-correlated bias (under-captures macrophages, over-captures small cells) and free dilation double-counts boundary pixels into two masks.

from skimage.segmentation import expand_labels, watershed

# expand_labels grows each label into background but stops at the midline between
# labels (no overlap), so each pixel is assigned once -- a partition, unlike free dilation
expanded = expand_labels(nuclear_masks, distance=3)   # ~3 px at 1 um; report the radius
assert expanded.max() == nuclear_masks.max()          # no cells created/destroyed

Per-Tool Failure Modes

Mesmer -- wrong image_mpp

Trigger: leaving the default mpp on 1 um IMC. Mechanism: Mesmer rescales the image to its 0.5 um training resolution; a wrong mpp rescales cells to the wrong learned size. Symptom: systematic over/under-segmentation across the whole image. Fix: pass image_mpp = true acquisition resolution (1.0 IMC, ~0.5 or finer MIBI).

Cellpose -- auto-diameter on tiny cells

Trigger: auto-diameter on ~5-px IMC nuclei. Mechanism: cyto rescales to a 30-px target; at single-digit diameters the rescale factor is large and unstable. Symptom: merged or fragmented masks. Fix: set diameter from known cell size in pixels, or use cpsam (no diameter dependence).

Generalist model -- wrong size prior

Trigger: default Cellpose cyto/cyto3 on IMC, trusted blindly. Mechanism: at ~5 px of evidence the learned PRIOR, not the image, draws the boundary, and a non-IMC prior is wrong. Symptom: plausible-looking but systematically biased masks. Fix: prefer Mesmer (TissueNet includes IMC/MIBI) or fine-tune Cellpose on the panel; evaluate on downstream proxies.

Channel compensation in the wrong order

Trigger: REDSEA before segmentation, or channel compensation after aggregation. Mechanism: channel spillover is pixel-level (must be corrected before the per-cell average); lateral spillover is defined on segmented neighbors (must be corrected after). Symptom: residual impossible co-expression. Fix: pixel-compensate -> segment -> aggregate -> REDSEA.

Quantitative Thresholds

Common Errors

References

• Giesen C, Wang HAO, Schapiro D, et al. 2014. Highly multiplexed imaging of tumor tissues with subcellular resolution by mass cytometry. Nat Methods 11(4):417-422. — IMC ~1 um resolution floor.
• Berg S, Kutra D, Kroeger T, et al. 2019. ilastik: interactive machine learning for (bio)image analysis. Nat Methods 16(12):1226-1232. — pixel classification stage.
• McQuin C, Goodman A, Chernyshev V, et al. 2018. CellProfiler 3.0: Next-generation image processing for biology. PLoS Biol 16(7):e2005970. — watershed instance segmentation.
• Schmidt U, Weigert M, Broaddus C, Myers G. 2018. Cell Detection with Star-Convex Polygons. MICCAI 2018, LNCS 11071:265-273. — StarDist nuclear baseline.
• Stringer C, Wang T, Michaelos M, Pachitariu M. 2021. Cellpose: a generalist algorithm for cellular segmentation. Nat Methods 18(1):100-106. — Cellpose diameters.
• Pachitariu M, Rariden M, Stringer C. 2025. Cellpose-SAM: superhuman generalization for cellular segmentation. bioRxiv doi:10.1101/2025.04.28.651001. — the cpsam SAM-backbone model (preprint).
• Greenwald NF, Miller G, Moen E, et al. 2022. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. Nat Biotechnol 40(4):555-565. — Mesmer/DeepCell, TissueNet, image_mpp.
• Bai Y, Zhu B, Rovira-Clave X, et al. 2021. Adjacent Cell Marker Lateral Spillover Compensation and Reinforcement for Multiplexed Images. Front Immunol 12:652631. — REDSEA boundary compensation; lateral spillover signature.
• Windhager J, Zanotelli VRT, Schulz D, et al. 2023. An end-to-end workflow for multiplexed image processing and analysis. Nat Protoc 18(11):3565-3613. — steinbock segmentation/measurement.

Related Skills

• data-preprocessing - channel spillover compensation precedes segmentation
• phenotyping - consumes the single-cell mask and intensities; double-positives diagnose segmentation
• spatial-analysis - over-segmentation corrupts neighborhood statistics
• quality-metrics - segmentation QC metrics and the impossible-co-expression monitor
• interactive-annotation - overlay masks on channels to audit boundaries