Estimates per-spot cell type composition of spatial transcriptomics mixtures (Visium, Slide-seq, Stereo-seq) from an scRNA-seq reference with cell2location, RCTD, SPOTlight, stereoscope, SpatialDWLS, or reference-free STdeconvolve. Use when deciding whether a platform even needs deconvolution (the resolution fork -- a 55um Visium spot is a 1-10-cell MIXTURE -> deconvolve, but a Xenium/MERFISH/CosMx cell is already single -> segment instead, and running deconvolution there invents fractions that do not exist); choosing cell2location (absolute abundance) vs RCTD/SPOTlight/stereoscope/SpatialDWLS (proportions only) by output and runtime; matching the scRNA reference to tissue and condition (the reference IS the result -- a missing cell type is silently misassigned to its nearest neighbor with no error flag); and handling compositional outputs that sum to 1 with CLR/ILR rather than naive per-type t-tests.
Install with Codex or Claude Copy this prompt, paste it into Codex, Claude, or another assistant, and let it review the skill page and install it for you.
A direct command skips the review prompt. Inspect the source before running it.
Estimates per-spot cell type composition of spatial transcriptomics mixtures (Visium, Slide-seq, Stereo-seq) from an scRNA-seq reference with cell2location, RCTD, SPOTlight, stereoscope, SpatialDWLS, or reference-free STdeconvolve. Use when deciding whether a platform even needs deconvolution (the resolution fork -- a 55um Visium spot is a 1-10-cell MIXTURE -> deconvolve, but a Xenium/MERFISH/CosMx cell is already single -> segment instead, and running deconvolution there invents fractions that do not exist); choosing cell2location (absolute abundance) vs RCTD/SPOTlight/stereoscope/SpatialDWLS (proportions only) by output and runtime; matching the scRNA reference to tissue and condition (the reference IS the result -- a missing cell type is silently misassigned to its nearest neighbor with no error flag); and handling compositional outputs that sum to 1 with CLR/ILR rather than naive per-type t-tests.
RCTD/SPOTlight/STdeconvolve are R packages (spacexr, SPOTlight, STdeconvolve via Bioconductor); call them from R or via rpy2.
Before using code patterns, verify installed versions match. If versions differ:
Python: pip show <package> then help(module.function) to check signatures
R: packageVersion('<pkg>') then ?function_name to verify parameters
If code throws ImportError, AttributeError, or TypeError, introspect the installed
package and adapt the example to match the actual API rather than retrying.
Spatial Deconvolution
"What cell types are in each spot?" -> Decompose a multi-cell capture spot into the fractions (or absolute numbers) of each cell type, using an annotated scRNA-seq reference as the basis.
The operation BRANCHES on the platform before any tool is chosen. The first question is not "which method" but "does this data even contain mixtures?" -- answered by the resolution fork below.
The resolution fork
Deconvolution recovers the cell-type composition of a MIXTURE. Whether a mixture exists is set by the capture unit size relative to a mammalian cell (~8-30um diameter), and it sorts every dataset into three regimes. Choosing the wrong regime is the most expensive error in spatial analysis -- far more damaging than picking the second-best method within a regime.
Platform
Unit size
Single cell?
Regime
Visium v1/v2
55um spot, 100um pitch
No (~1-10+ cells/spot)
DECONVOLVE
GeoMx DSP
region of interest
No (many cells)
DECONVOLVE (SpatialDecon)
Visium HD
2um bins, analyzed at 8/16um
8um still mixes cells
AMBIGUOUS (reconstruct OR deconvolve)
Stereo-seq
~220nm spots, binned (bin50/bin100)
binned to cell scale
AMBIGUOUS
Slide-seqV2
10um beads
near single-cell
AMBIGUOUS (RCTD doublet-mode common)
Xenium
transcript point cloud + DAPI
YES (segmented)
SEGMENT + annotate -- do NOT deconvolve
MERFISH / MERSCOPE
subcellular
YES
SEGMENT + annotate -- do NOT deconvolve
CosMx SMI
subcellular
YES
SEGMENT + annotate -- do NOT deconvolve
DECONVOLVE: a capture array has no per-transcript cell assignment, so the spot is an unavoidable mixture and only mixture modeling recovers composition. The H&E underlay can count nuclei but cannot say which transcript came from which cell.
SEGMENT (imaging platforms): the data are already single cells. The work is segmentation (assign transcripts to cells) then annotation (cluster + markers, or label-transfer). Running deconvolution here is conceptually WRONG -- it fabricates fractional mixtures inside cells that are already pure. See image-analysis for segmentation and single-cell/cell-annotation for label transfer.
AMBIGUOUS (the near-single-cell middle): bins/beads still hold partial or multiple cells. When a high-quality registered image exists (Visium HD), morphology-driven cell RECONSTRUCTION (Bin2cell, StarDist/Cellpose nuclei expansion) is increasingly preferred over treating bins as fixed mixtures; without per-bead morphology (Slide-seqV2), assignment/deconvolution (often RCTD doublet-mode) remains standard. No settled consensus -- see high-resolution-binning.
Governing Principle
A deconvolution result is a PROJECTION of the single-cell reference onto the spatial data. The reference is not a neutral input -- it is the dominant determinant of the answer, more than the algorithm.
The #1 trap is the missing or mismatched cell type. If a real type is ABSENT from the reference, its transcripts are forced onto the transcriptionally nearest type that IS present. The proportions still sum to 1 and look clean, and there is no internal warning -- garbage reference produces confident garbage proportions. The reference must match the TISSUE, the CONDITION (a healthy reference mis-estimates activated-immune or malignant states whose expression has shifted), and ideally the technology/protocol (snRNA-seq vs whole-cell dissociation bias propagates straight into the numbers). The reference is NOT ground truth.
Every benchmark reinforces the same lesson: reference quality and the cell-type abundance pattern swing results MORE than method choice, and a plain NNLS baseline outperforms nearly half the dedicated methods (Sang-Aram 2024 eLife 12:RP88431; Li 2022 Nat Methods 19:662-670). Method-shopping is the wrong lever; reference quality is the right one. Rare-type fractions (below a few percent) are the least reliable numbers in the output -- corroborate any rare type with its spatial marker genes before believing it. More tools is not more confidence: two NB-regression methods agreeing is pseudo-replication, not orthogonal validation. The defensible confidence move is reference-sensitivity analysis (perturb or swap the reference) plus an orthogonal modality.
Outputs are COMPOSITIONAL: per-spot proportions live on a simplex (sum to 1), so an increase in one type mechanically decreases the others. Downstream comparison must use CLR/ILR or compositional tests -- naive per-type t-tests/correlations on raw proportions are mis-specified.
Choosing a method
The first axis is output: cell2location uniquely returns ABSOLUTE cell abundance (expected cell numbers per spot); the rest return proportions only. "Number of cells of type X" and "fraction of the spot that is type X" answer different biological questions. The second axis is runtime and whether spatial coordinates or a platform-shift correction are modeled. When competing methods exist, verify current best practice against the latest benchmark before committing -- this field moves fast.
Method
Model
Reference
Output
Best when
Fails when
cell2location
Bayesian hierarchical NB (variational, pyro/scvi-tools)
yes
ABSOLUTE abundance
absolute counts wanted; large atlases; models platform shift
fast, widely used; doublet-mode for Slide-seq/high-res
non-R pipelines without rpy2
stereoscope
Negative-binomial MLE of spot mixtures
yes
proportions
principled NB; in scvi-tools
speed (among slowest)
SPOTlight
Seeded NMF + NNLS
yes
proportions
fast, transparent, R/Bioconductor
mid-pack accuracy
SpatialDWLS
Dampened weighted least squares + marker enrichment
yes
proportions
fastest tier; consistently top in Li 2022
needs Giotto
CARD
CAR-prior spatially-informed NMF regression
yes (can run ref-free)
proportions (smoothed)
spatially structured tissue; coordinates help
sharp composition boundaries (over-smooths)
Tangram
Deep-learning alignment (cell->voxel mapping)
yes
mapping (proportions as by-product)
transcript imputation; flexible platforms
pure proportion accuracy (it is a mapper)
STdeconvolve
Reference-FREE LDA topic model
NO
proportions + topic profiles
no matched reference exists; sanity check
types co-occurring in fixed ratios; topics need post-hoc annotation
cell2location and RCTD are reliably top-tier across independent benchmarks (Li 2022; Sang-Aram 2024; Nat Commun 2023 14:1548). When no matched reference exists, STdeconvolve is the escape hatch -- it is immune to the missing-type trap because it uses no reference, but its topics are de novo and must be annotated afterward.
cell2location step 1: reference signatures
Goal: Learn each cell type's per-gene expression signature from the annotated scRNA-seq reference, correcting for the technical/batch structure of the reference.
Approach: Filter genes to informative ones, fit a negative-binomial RegressionModel with the cell-type label and any batch as covariates, then export the posterior per-cluster mean expression. cell2location consumes RAW integer counts -- do NOT pass log-normalized data.
import cell2location
import numpy as np
import scanpy as sc
from cell2location.utils.filtering import filter_genes
from cell2location.models import RegressionModel
adata_ref = sc.read_h5ad('reference_scrna.h5ad') # raw counts in .X
adata_ref.obs['cell_type'] = adata_ref.obs['cell_type'].astype('category')
selected = filter_genes(adata_ref, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)
adata_ref = adata_ref[:, selected].copy()
# batch_key absorbs technical structure across reference samples; drop if a single batch
RegressionModel.setup_anndata(adata_ref, labels_key='cell_type', batch_key='sample')
mod = RegressionModel(adata_ref)
mod.train(max_epochs=250, accelerator='gpu') # use_gpu= is deprecated; accelerator in {'gpu','cpu','auto'}
adata_ref = mod.export_posterior(adata_ref, sample_kwargs={'num_samples': 1000})
factors = adata_ref.uns['mod']['factor_names']
if'means_per_cluster_mu_fg'in adata_ref.varm:
inf_aver = adata_ref.varm['means_per_cluster_mu_fg'][[f'means_per_cluster_mu_fg_{f}'for f in factors]].copy()
else:
inf_aver = adata_ref.var[[f'means_per_cluster_mu_fg_{f}'for f in factors]].copy()
inf_aver.columns = factors # genes x cell_types signature matrix
cell2location step 2: spatial mapping
Goal: Decompose each spatial spot into absolute cell-type abundances using the reference signatures.
Approach: Restrict both objects to shared genes, set up the Cell2location model with the signature matrix and the expected cells-per-spot, then train. N_cells_per_location is a tissue-dependent prior (Visium ~10-30, denser tissue higher); detection_alpha controls within-experiment normalization (20 is the tutorial default; raise toward 200 if technical variability in total counts is high).
adata_vis = sc.read_h5ad('visium.h5ad') # raw counts
shared = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, shared].copy()
inf_aver = inf_aver.loc[shared, :]
cell2location.models.Cell2location.setup_anndata(adata_vis, batch_key='sample')
mod_sp = cell2location.models.Cell2location(adata_vis, cell_state_df=inf_aver,
N_cells_per_location=30, detection_alpha=20)
mod_sp.train(max_epochs=30000, batch_size=None, train_size=1, accelerator='gpu')
adata_vis = mod_sp.export_posterior(adata_vis, sample_kwargs={'num_samples': 1000, 'batch_size': mod_sp.adata.n_obs})
# q05 = 5% posterior quantile = 'at least this many cells of this type are present' (conservative)
abundance = adata_vis.obsm['q05_cell_abundance_w_sf'] # ABSOLUTE expected cell numbers per spot
abundance.columns = factors
adata_vis.obs[abundance.columns] = abundance.values
cell2location returns absolute abundances. Convert to proportions only if proportions are the question -- doing so discards the information that distinguishes cell2location from the proportion-only methods.
Handling compositional outputs
Goal: Compare cell-type composition across spots, regions, or conditions without the spurious negative correlations that the sum-to-1 constraint manufactures.
Approach: Map proportions out of the simplex with the centered log-ratio (CLR) before any correlation, t-test, or linear model. Add a small pseudocount because CLR is undefined at zero.
import numpy as np
defclr(proportions, pseudocount=1e-6):
p = proportions + pseudocount
p = p / p.sum(axis=1, keepdims=True)
log_p = np.log(p)
return log_p - log_p.mean(axis=1, keepdims=True) # subtract per-spot geometric-mean log
abund = adata_vis.obsm['q05_cell_abundance_w_sf'].values
proportions = abund / abund.sum(axis=1, keepdims=True)
clr_comp = clr(proportions) # now safe for downstream correlation / DE / t-tests
For differential abundance between conditions, prefer a compositional method (ALDEx2, scCODA, or a Dirichlet-multinomial model) over per-type t-tests on raw fractions; the same closed-data logic that breaks naive correlations breaks naive DA.
Reference-free sanity check
Goal: Detect a missing-reference-type problem and recover composition when no matched scRNA-seq reference exists.
Approach: Run STdeconvolve (LDA topic model, R) on the spatial data alone, then check whether any de novo topic matches no reference cell type -- a topic with strong spatial structure and marker genes for a type absent from the reference is direct evidence the reference is incomplete.
library(STdeconvolve)
counts <- cleanCounts(spatial_counts_matrix, min.lib.size =100)# genes x spots
corpus <- restrictCorpus(counts, removeAbove =1.0, removeBelow =0.05)
ldas <- fitLDA(t(as.matrix(corpus)), Ks = seq(8,15))# K = number of topics, scan a range
opt <- optimalModel(models = ldas, opt ='min')
res <- getBetaTheta(opt, perc.filt =0.05)
deconv_prop <- res$theta # spots x topics proportions# annotate topics post hoc via res$beta (topic gene profiles) against known markers
Validating against marker genes
Goal: Confirm that an estimated cell-type fraction tracks the in-situ expression of that type's canonical markers, not the reference's wishful thinking.
Approach: For each type, correlate its estimated proportion across spots with the mean spatial expression of its marker genes; weak or negative correlation flags a misassignment or a reference mismatch.
markers = {'T_cell': ['CD3D', 'CD3E', 'CD8A'], 'Macrophage': ['CD68', 'CD14', 'CSF1R'], 'Epithelial': ['EPCAM', 'KRT8']}
for ct, genes in markers.items():
present = [g for g in genes if g in adata_vis.var_names]
ifnot present or ct notin proportions_df.columns:
continue
expr = np.asarray(adata_vis[:, present].X.mean(axis=1)).ravel()
corr = np.corrcoef(expr, proportions_df[ct].values)[0, 1]
print(f'{ct}: marker-vs-proportion r = {corr:.3f}') # low r -> suspect reference or misassignment
Common Errors
Symptom
Cause
Fix
Deconvolution "works" on Xenium/MERFISH/CosMx but fractions are nonsensical
Ran deconvolution on single-cell-resolution imaging data, inventing mixtures inside pure cells
Segment then annotate (image-analysis, single-cell/cell-annotation); deconvolution does not apply
A histologically present cell type appears in NO spot
Type is absent from the reference; its signal was silently reassigned to the nearest present type
Add the missing type to the reference; cross-check with STdeconvolve for an unmatched topic
Confident proportions, but disease/activated states look wrong
RCTD doublet-mode or spillover-aware segmentation; treat ubiquitous low fractions with suspicion
Related Skills
spatial-domains - group spots into tissue regions; a domain is a region, not a cell type or a niche
high-resolution-binning - the AMBIGUOUS regime (Visium HD, Stereo-seq, Slide-seq): reconstruct cells vs deconvolve bins
image-analysis - segment cells from imaging platforms, where deconvolution does not apply
single-cell/cell-annotation - annotate the imaging cells after segmentation, and label the scRNA-seq reference
single-cell/preprocessing - build a clean reference; the reference IS the result
spatial-preprocessing - QC and normalize the spatial data before deconvolution
spatial-visualization - map estimated proportions and abundances onto the tissue
References
Kleshchevnikov V, Shmatko A, Dann E, et al. (2022) Cell2location maps fine-grained cell types in spatial transcriptomics. Nature Biotechnology 40:661-671. DOI 10.1038/s41587-021-01139-4
Cable DM, Murray E, Zou LS, et al. (2022) Robust decomposition of cell type mixtures in spatial transcriptomics (RCTD). Nature Biotechnology 40:517-526. DOI 10.1038/s41587-021-00830-w
Andersson A, Bergenstrahle J, Asp M, et al. (2020) Single-cell and spatial transcriptomics enables probabilistic inference of cell type topography (stereoscope). Communications Biology 3:565. DOI 10.1038/s42003-020-01247-y
Elosua-Bayes M, Nieto P, Mereu E, Gut I, Heyn H (2021) SPOTlight: seeded NMF regression to deconvolute spatial transcriptomics spots with single-cell transcriptomes. Nucleic Acids Research 49(9):e50. DOI 10.1093/nar/gkab043
Biancalani T, Scalia G, Buffoni L, et al. (2021) Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram. Nature Methods 18(11):1352-1362. DOI 10.1038/s41592-021-01264-7
Ma Y, Zhou X (2022) Spatially informed cell-type deconvolution for spatial transcriptomics (CARD). Nature Biotechnology 40:1349-1359. DOI 10.1038/s41587-022-01273-7
Dong R, Yuan GC (2021) SpatialDWLS: accurate deconvolution of spatial transcriptomic data. Genome Biology 22:145. DOI 10.1186/s13059-021-02362-7
Miller BF, Huang F, Atta L, Sahoo A, Fan J (2022) Reference-free cell type deconvolution of multi-cellular pixel-resolution spatially resolved transcriptomics data (STdeconvolve). Nature Communications 13:2339. DOI 10.1038/s41467-022-30033-z
Sang-Aram C, Browaeys R, Seurinck R, Saeys Y (2024) Spotless, a reproducible pipeline for benchmarking cell type deconvolution in spatial transcriptomics. eLife 12:RP88431. DOI 10.7554/eLife.88431
Li B, Zhang W, Guo C, et al. (2022) Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nature Methods 19:662-670. DOI 10.1038/s41592-022-01480-9
