bio-spatial-transcriptomics-spatial-deconvolution

name	bio-spatial-transcriptomics-spatial-deconvolution
description	Estimates per-spot cell type composition of spatial transcriptomics mixtures (Visium, Slide-seq, Stereo-seq) from an scRNA-seq reference with cell2location, RCTD, SPOTlight, stereoscope, SpatialDWLS, or reference-free STdeconvolve. Use when deciding whether a platform even needs deconvolution (the resolution fork -- a 55um Visium spot is a 1-10-cell MIXTURE -> deconvolve, but a Xenium/MERFISH/CosMx cell is already single -> segment instead, and running deconvolution there invents fractions that do not exist); choosing cell2location (absolute abundance) vs RCTD/SPOTlight/stereoscope/SpatialDWLS (proportions only) by output and runtime; matching the scRNA reference to tissue and condition (the reference IS the result -- a missing cell type is silently misassigned to its nearest neighbor with no error flag); and handling compositional outputs that sum to 1 with CLR/ILR rather than naive per-type t-tests.
tool_type	python
primary_tool	cell2location

Version Compatibility

Reference examples tested with: cell2location 0.1.4+, scvi-tools 1.0+, scanpy 1.10+, anndata 0.10+, numpy 1.26+

RCTD/SPOTlight/STdeconvolve are R packages (spacexr, SPOTlight, STdeconvolve via Bioconductor); call them from R or via rpy2.

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• R: packageVersion('<pkg>') then ?function_name to verify parameters

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Spatial Deconvolution

"What cell types are in each spot?" -> Decompose a multi-cell capture spot into the fractions (or absolute numbers) of each cell type, using an annotated scRNA-seq reference as the basis.

• Python: cell2location (RegressionModel -> Cell2location), stereoscope/DestVI (scvi-tools), Tangram (tg.map_cells_to_space), STdeconvolve (reference-free, R)
• R: RCTD (spacexr::create.RCTD/run.RCTD), SPOTlight, SpatialDWLS (Giotto), CARD

The operation BRANCHES on the platform before any tool is chosen. The first question is not "which method" but "does this data even contain mixtures?" -- answered by the resolution fork below.

The resolution fork

Deconvolution recovers the cell-type composition of a MIXTURE. Whether a mixture exists is set by the capture unit size relative to a mammalian cell (~8-30um diameter), and it sorts every dataset into three regimes. Choosing the wrong regime is the most expensive error in spatial analysis -- far more damaging than picking the second-best method within a regime.

Platform	Unit size	Single cell?	Regime
Visium v1/v2	55um spot, 100um pitch	No (~1-10+ cells/spot)	DECONVOLVE
GeoMx DSP	region of interest	No (many cells)	DECONVOLVE (SpatialDecon)
Visium HD	2um bins, analyzed at 8/16um	8um still mixes cells	AMBIGUOUS (reconstruct OR deconvolve)
Stereo-seq	~220nm spots, binned (bin50/bin100)	binned to cell scale	AMBIGUOUS
Slide-seqV2	10um beads	near single-cell	AMBIGUOUS (RCTD doublet-mode common)
Xenium	transcript point cloud + DAPI	YES (segmented)	SEGMENT + annotate -- do NOT deconvolve
MERFISH / MERSCOPE	subcellular	YES	SEGMENT + annotate -- do NOT deconvolve
CosMx SMI	subcellular	YES	SEGMENT + annotate -- do NOT deconvolve

• DECONVOLVE: a capture array has no per-transcript cell assignment, so the spot is an unavoidable mixture and only mixture modeling recovers composition. The H&E underlay can count nuclei but cannot say which transcript came from which cell.
• SEGMENT (imaging platforms): the data are already single cells. The work is segmentation (assign transcripts to cells) then annotation (cluster + markers, or label-transfer). Running deconvolution here is conceptually WRONG -- it fabricates fractional mixtures inside cells that are already pure. See image-analysis for segmentation and single-cell/cell-annotation for label transfer.
• AMBIGUOUS (the near-single-cell middle): bins/beads still hold partial or multiple cells. When a high-quality registered image exists (Visium HD), morphology-driven cell RECONSTRUCTION (Bin2cell, StarDist/Cellpose nuclei expansion) is increasingly preferred over treating bins as fixed mixtures; without per-bead morphology (Slide-seqV2), assignment/deconvolution (often RCTD doublet-mode) remains standard. No settled consensus -- see high-resolution-binning.

Governing Principle

A deconvolution result is a PROJECTION of the single-cell reference onto the spatial data. The reference is not a neutral input -- it is the dominant determinant of the answer, more than the algorithm.

The #1 trap is the missing or mismatched cell type. If a real type is ABSENT from the reference, its transcripts are forced onto the transcriptionally nearest type that IS present. The proportions still sum to 1 and look clean, and there is no internal warning -- garbage reference produces confident garbage proportions. The reference must match the TISSUE, the CONDITION (a healthy reference mis-estimates activated-immune or malignant states whose expression has shifted), and ideally the technology/protocol (snRNA-seq vs whole-cell dissociation bias propagates straight into the numbers). The reference is NOT ground truth.

Every benchmark reinforces the same lesson: reference quality and the cell-type abundance pattern swing results MORE than method choice, and a plain NNLS baseline outperforms nearly half the dedicated methods (Sang-Aram 2024 eLife 12:RP88431; Li 2022 Nat Methods 19:662-670). Method-shopping is the wrong lever; reference quality is the right one. Rare-type fractions (below a few percent) are the least reliable numbers in the output -- corroborate any rare type with its spatial marker genes before believing it. More tools is not more confidence: two NB-regression methods agreeing is pseudo-replication, not orthogonal validation. The defensible confidence move is reference-sensitivity analysis (perturb or swap the reference) plus an orthogonal modality.

Outputs are COMPOSITIONAL: per-spot proportions live on a simplex (sum to 1), so an increase in one type mechanically decreases the others. Downstream comparison must use CLR/ILR or compositional tests -- naive per-type t-tests/correlations on raw proportions are mis-specified.

Choosing a method

The first axis is output: cell2location uniquely returns ABSOLUTE cell abundance (expected cell numbers per spot); the rest return proportions only. "Number of cells of type X" and "fraction of the spot that is type X" answer different biological questions. The second axis is runtime and whether spatial coordinates or a platform-shift correction are modeled. When competing methods exist, verify current best practice against the latest benchmark before committing -- this field moves fast.

Method	Model	Reference	Output	Best when	Fails when
cell2location	Bayesian hierarchical NB (variational, pyro/scvi-tools)	yes	ABSOLUTE abundance	absolute counts wanted; large atlases; models platform shift	GPU-light setups (slow VI); over-trusting rare types
RCTD (spacexr)	Poisson + per-gene platform-effect random effect	yes	proportions	fast, widely used; doublet-mode for Slide-seq/high-res	non-R pipelines without rpy2
stereoscope	Negative-binomial MLE of spot mixtures	yes	proportions	principled NB; in scvi-tools	speed (among slowest)
SPOTlight	Seeded NMF + NNLS	yes	proportions	fast, transparent, R/Bioconductor	mid-pack accuracy
SpatialDWLS	Dampened weighted least squares + marker enrichment	yes	proportions	fastest tier; consistently top in Li 2022	needs Giotto
CARD	CAR-prior spatially-informed NMF regression	yes (can run ref-free)	proportions (smoothed)	spatially structured tissue; coordinates help	sharp composition boundaries (over-smooths)
Tangram	Deep-learning alignment (cell->voxel mapping)	yes	mapping (proportions as by-product)	transcript imputation; flexible platforms	pure proportion accuracy (it is a mapper)
STdeconvolve	Reference-FREE LDA topic model	NO	proportions + topic profiles	no matched reference exists; sanity check	types co-occurring in fixed ratios; topics need post-hoc annotation

cell2location and RCTD are reliably top-tier across independent benchmarks (Li 2022; Sang-Aram 2024; Nat Commun 2023 14:1548). When no matched reference exists, STdeconvolve is the escape hatch -- it is immune to the missing-type trap because it uses no reference, but its topics are de novo and must be annotated afterward.

cell2location step 1: reference signatures

Goal: Learn each cell type's per-gene expression signature from the annotated scRNA-seq reference, correcting for the technical/batch structure of the reference.

Approach: Filter genes to informative ones, fit a negative-binomial RegressionModel with the cell-type label and any batch as covariates, then export the posterior per-cluster mean expression. cell2location consumes RAW integer counts -- do NOT pass log-normalized data.

import cell2location
import numpy as np
import scanpy as sc
from cell2location.utils.filtering import filter_genes
from cell2location.models import RegressionModel

adata_ref = sc.read_h5ad('reference_scrna.h5ad')           # raw counts in .X
adata_ref.obs['cell_type'] = adata_ref.obs['cell_type'].astype('category')

selected = filter_genes(adata_ref, cell_count_cutoff=5, cell_percentage_cutoff2=0.03, nonz_mean_cutoff=1.12)
adata_ref = adata_ref[:, selected].copy()

# batch_key absorbs technical structure across reference samples; drop if a single batch
RegressionModel.setup_anndata(adata_ref, labels_key='cell_type', batch_key='sample')
mod = RegressionModel(adata_ref)
mod.train(max_epochs=250, accelerator='gpu')               # use_gpu= is deprecated; accelerator in {'gpu','cpu','auto'}
adata_ref = mod.export_posterior(adata_ref, sample_kwargs={'num_samples': 1000})

factors = adata_ref.uns['mod']['factor_names']
if 'means_per_cluster_mu_fg' in adata_ref.varm:
    inf_aver = adata_ref.varm['means_per_cluster_mu_fg'][[f'means_per_cluster_mu_fg_{f}' for f in factors]].copy()
else:
    inf_aver = adata_ref.var[[f'means_per_cluster_mu_fg_{f}' for f in factors]].copy()
inf_aver.columns = factors                                 # genes x cell_types signature matrix

cell2location step 2: spatial mapping

Goal: Decompose each spatial spot into absolute cell-type abundances using the reference signatures.

Approach: Restrict both objects to shared genes, set up the Cell2location model with the signature matrix and the expected cells-per-spot, then train. N_cells_per_location is a tissue-dependent prior (Visium ~10-30, denser tissue higher); detection_alpha controls within-experiment normalization (20 is the tutorial default; raise toward 200 if technical variability in total counts is high).

adata_vis = sc.read_h5ad('visium.h5ad')                    # raw counts
shared = np.intersect1d(adata_vis.var_names, inf_aver.index)
adata_vis = adata_vis[:, shared].copy()
inf_aver = inf_aver.loc[shared, :]

cell2location.models.Cell2location.setup_anndata(adata_vis, batch_key='sample')
mod_sp = cell2location.models.Cell2location(adata_vis, cell_state_df=inf_aver,
                                            N_cells_per_location=30, detection_alpha=20)
mod_sp.train(max_epochs=30000, batch_size=None, train_size=1, accelerator='gpu')
adata_vis = mod_sp.export_posterior(adata_vis, sample_kwargs={'num_samples': 1000, 'batch_size': mod_sp.adata.n_obs})

# q05 = 5% posterior quantile = 'at least this many cells of this type are present' (conservative)
abundance = adata_vis.obsm['q05_cell_abundance_w_sf']      # ABSOLUTE expected cell numbers per spot
abundance.columns = factors
adata_vis.obs[abundance.columns] = abundance.values

cell2location returns absolute abundances. Convert to proportions only if proportions are the question -- doing so discards the information that distinguishes cell2location from the proportion-only methods.

Handling compositional outputs

Goal: Compare cell-type composition across spots, regions, or conditions without the spurious negative correlations that the sum-to-1 constraint manufactures.

Approach: Map proportions out of the simplex with the centered log-ratio (CLR) before any correlation, t-test, or linear model. Add a small pseudocount because CLR is undefined at zero.

import numpy as np

def clr(proportions, pseudocount=1e-6):
    p = proportions + pseudocount
    p = p / p.sum(axis=1, keepdims=True)
    log_p = np.log(p)
    return log_p - log_p.mean(axis=1, keepdims=True)       # subtract per-spot geometric-mean log

abund = adata_vis.obsm['q05_cell_abundance_w_sf'].values
proportions = abund / abund.sum(axis=1, keepdims=True)
clr_comp = clr(proportions)                                # now safe for downstream correlation / DE / t-tests

For differential abundance between conditions, prefer a compositional method (ALDEx2, scCODA, or a Dirichlet-multinomial model) over per-type t-tests on raw fractions; the same closed-data logic that breaks naive correlations breaks naive DA.

Reference-free sanity check

Goal: Detect a missing-reference-type problem and recover composition when no matched scRNA-seq reference exists.

Approach: Run STdeconvolve (LDA topic model, R) on the spatial data alone, then check whether any de novo topic matches no reference cell type -- a topic with strong spatial structure and marker genes for a type absent from the reference is direct evidence the reference is incomplete.

library(STdeconvolve)
counts <- cleanCounts(spatial_counts_matrix, min.lib.size = 100)   # genes x spots
corpus <- restrictCorpus(counts, removeAbove = 1.0, removeBelow = 0.05)
ldas <- fitLDA(t(as.matrix(corpus)), Ks = seq(8, 15))              # K = number of topics, scan a range
opt <- optimalModel(models = ldas, opt = 'min')
res <- getBetaTheta(opt, perc.filt = 0.05)
deconv_prop <- res$theta                                          # spots x topics proportions
# annotate topics post hoc via res$beta (topic gene profiles) against known markers

Validating against marker genes

Goal: Confirm that an estimated cell-type fraction tracks the in-situ expression of that type's canonical markers, not the reference's wishful thinking.

Approach: For each type, correlate its estimated proportion across spots with the mean spatial expression of its marker genes; weak or negative correlation flags a misassignment or a reference mismatch.

markers = {'T_cell': ['CD3D', 'CD3E', 'CD8A'], 'Macrophage': ['CD68', 'CD14', 'CSF1R'], 'Epithelial': ['EPCAM', 'KRT8']}
for ct, genes in markers.items():
    present = [g for g in genes if g in adata_vis.var_names]
    if not present or ct not in proportions_df.columns:
        continue
    expr = np.asarray(adata_vis[:, present].X.mean(axis=1)).ravel()
    corr = np.corrcoef(expr, proportions_df[ct].values)[0, 1]
    print(f'{ct}: marker-vs-proportion r = {corr:.3f}')        # low r -> suspect reference or misassignment

Common Errors

Symptom	Cause	Fix
Deconvolution "works" on Xenium/MERFISH/CosMx but fractions are nonsensical	Ran deconvolution on single-cell-resolution imaging data, inventing mixtures inside pure cells	Segment then annotate (image-analysis, single-cell/cell-annotation); deconvolution does not apply
A histologically present cell type appears in NO spot	Type is absent from the reference; its signal was silently reassigned to the nearest present type	Add the missing type to the reference; cross-check with STdeconvolve for an unmatched topic
Confident proportions, but disease/activated states look wrong	Condition mismatch -- healthy reference deconvolving diseased tissue	Use a condition-matched reference; activated/malignant states are transcriptionally far from healthy
Per-type t-test finds "everything changes oppositely"	Naive test on compositional (sum-to-1) proportions creates spurious negative correlation	CLR/ILR transform first; use ALDEx2/scCODA/Dirichlet-multinomial for differential abundance
Rare cell type fraction swings wildly between runs/references	Rare-type fractions are the least reliable output; abundance-pattern sensitivity	Treat <few-percent fractions skeptically; corroborate with spatial markers; do reference-sensitivity analysis
ValueError/garbage from cell2location after normalizing	Passed log-normalized data; cell2location needs RAW integer counts	Feed raw counts; stash normalized layers separately
TypeError: unexpected keyword 'use_gpu'	use_gpu deprecated in current scvi-tools	Use accelerator='gpu' (or 'cpu'/'auto')
Two methods agree, reported as validation	Two NB-regression methods are pseudo-replication, not orthogonal	Validate by perturbing the reference and against an orthogonal modality (matched imaging, in-situ markers)
Every spot contains a little of every immune type	Spot-edge transcript spillover / diffusion inflates apparent co-localization	RCTD doublet-mode or spillover-aware segmentation; treat ubiquitous low fractions with suspicion

Related Skills

• spatial-domains - group spots into tissue regions; a domain is a region, not a cell type or a niche
• high-resolution-binning - the AMBIGUOUS regime (Visium HD, Stereo-seq, Slide-seq): reconstruct cells vs deconvolve bins
• image-analysis - segment cells from imaging platforms, where deconvolution does not apply
• single-cell/cell-annotation - annotate the imaging cells after segmentation, and label the scRNA-seq reference
• single-cell/preprocessing - build a clean reference; the reference IS the result
• spatial-preprocessing - QC and normalize the spatial data before deconvolution
• spatial-visualization - map estimated proportions and abundances onto the tissue

References

• Kleshchevnikov V, Shmatko A, Dann E, et al. (2022) Cell2location maps fine-grained cell types in spatial transcriptomics. Nature Biotechnology 40:661-671. DOI 10.1038/s41587-021-01139-4
• Cable DM, Murray E, Zou LS, et al. (2022) Robust decomposition of cell type mixtures in spatial transcriptomics (RCTD). Nature Biotechnology 40:517-526. DOI 10.1038/s41587-021-00830-w
• Andersson A, Bergenstrahle J, Asp M, et al. (2020) Single-cell and spatial transcriptomics enables probabilistic inference of cell type topography (stereoscope). Communications Biology 3:565. DOI 10.1038/s42003-020-01247-y
• Elosua-Bayes M, Nieto P, Mereu E, Gut I, Heyn H (2021) SPOTlight: seeded NMF regression to deconvolute spatial transcriptomics spots with single-cell transcriptomes. Nucleic Acids Research 49(9):e50. DOI 10.1093/nar/gkab043
• Biancalani T, Scalia G, Buffoni L, et al. (2021) Deep learning and alignment of spatially resolved single-cell transcriptomes with Tangram. Nature Methods 18(11):1352-1362. DOI 10.1038/s41592-021-01264-7
• Ma Y, Zhou X (2022) Spatially informed cell-type deconvolution for spatial transcriptomics (CARD). Nature Biotechnology 40:1349-1359. DOI 10.1038/s41587-022-01273-7
• Dong R, Yuan GC (2021) SpatialDWLS: accurate deconvolution of spatial transcriptomic data. Genome Biology 22:145. DOI 10.1186/s13059-021-02362-7
• Miller BF, Huang F, Atta L, Sahoo A, Fan J (2022) Reference-free cell type deconvolution of multi-cellular pixel-resolution spatially resolved transcriptomics data (STdeconvolve). Nature Communications 13:2339. DOI 10.1038/s41467-022-30033-z
• Sang-Aram C, Browaeys R, Seurinck R, Saeys Y (2024) Spotless, a reproducible pipeline for benchmarking cell type deconvolution in spatial transcriptomics. eLife 12:RP88431. DOI 10.7554/eLife.88431
• Li B, Zhang W, Guo C, et al. (2022) Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution. Nature Methods 19:662-670. DOI 10.1038/s41592-022-01480-9