bio-spatial-transcriptomics-spatial-domains

name	bio-spatial-transcriptomics-spatial-domains
description	Identify spatially coherent tissue domains (regions like cortical layers, tumor vs stroma) in Visium, Visium HD, Xenium, MERFISH, Slide-seq, and Stereo-seq data with Squidpy, BANKSY, BayesSpace, STAGATE, and GraphST. Use when distinguishing a domain (a region with many cell types) from a cell type (one cell's identity) and a niche (local cell-type composition); choosing a domain method by tissue geometry (laminar/continuous vs high-resolution imaging vs non-contiguous); tuning the spatial-weight knob (BANKSY lambda, BayesSpace smoothing, SpaGCN histology weight, GNN graph radius) to avoid over-smoothing into blobs or under-smoothing into salt-and-pepper; choosing the number of domains k as a biological decision with k+-1 sensitivity; and reading the Yuan 2024 benchmark with the DLPFC continuous-laminar caveat.
tool_type	python
primary_tool	squidpy

Version Compatibility

Reference examples tested with: squidpy 1.4+, scanpy 1.10+, anndata 0.10+, scikit-learn 1.4+; BANKSY (banksy_py / R Banksy), BayesSpace 1.12+ (R), STAGATE/GraphST optional

Before using code patterns, verify installed versions match. If versions differ:

• Python: pip show <package> then help(module.function) to check signatures
• R: packageVersion('<pkg>') then ?function_name to verify parameters

If code throws ImportError, AttributeError, or TypeError, introspect the installed package and adapt the example to match the actual API rather than retrying.

Spatial Domain Detection

"Identify tissue domains in my section" -> Partition the tissue into spatially contiguous regions of homogeneous expression/composition, using BOTH transcriptional similarity AND spatial proximity, so the output is a region label per spot/cell that is spatially coherent.

• Python: squidpy.gr.spatial_neighbors for the graph, then a neighbor-augmented or graph-based domain method (BANKSY, STAGATE, GraphST)
• R: BayesSpace (spatialCluster), BASS, or Banksy

Governing Principle

A spatial domain is a REGION, not a cell type, and not a niche -- conflating the three is the central conceptual error of this analysis. A CELL TYPE is one cell's transcriptional identity. A NICHE (cellular neighborhood) is the local cell-type COMPOSITION around a cell -- which types co-occur. A SPATIAL DOMAIN is a contiguous tissue region (a cortical layer, tumor core, stromal band) that contains MANY cell types and several niches. If the question is "which cell types co-occur," that is a niche question and belongs to neighborhood-enrichment analysis (spatial-statistics), NOT domain segmentation; running the wrong one answers the wrong question.

Domain methods exist because plain Leiden/Louvain on expression alone ignores coordinates and produces salt-and-pepper, spatially-incoherent labels. Every domain method deliberately adds a spatial term so neighbors tend to share a label. The load-bearing decision is therefore NOT the clustering algorithm -- it is the SPATIAL-WEIGHT knob (BANKSY lambda, BayesSpace smoothing, SpaGCN histology weight, GNN graph radius). Too much spatial weight is the #1 trap, over-smoothing: it erases real boundaries and merges biologically distinct regions into blobs. Too little reverts to salt-and-pepper. The number of domains k is the second decision, and it is a BIOLOGICAL choice (how fine a regionalization the question needs), not a statistic to optimize against a silhouette score -- report k, justify it, and show sensitivity to k+-1.

Domain vs Niche vs Cell Type

Concept	What it is	Unit	Right tool
Cell type	One cell's transcriptional identity (lineage/state)	a cell	clustering + markers (single-cell/clustering)
Niche / cellular neighborhood	Local cell-type composition -- which types co-occur around a cell	a neighborhood	neighborhood enrichment / co-occurrence (spatial-statistics)
Spatial domain	Contiguous tissue region of homogeneous expression, containing many types	a region	domain segmentation (this skill)

A domain can contain several niches; a niche can span domain boundaries. BANKSY and BASS switch between cell-typing and domain detection via a single parameter, which underscores that these are different outputs of related machinery, not the same task.

Domain Methods by Mechanism

The mechanism for "using the neighbors" is the axis that separates the methods. No method is a universal winner (Yuan 2024 Nat Methods); selection is scenario-specific.

Method	Spatial mechanism	Needs k?	Best when	Fails when
BayesSpace	MRF (Potts) smoothing prior on a t-mixture in PCA space; also enhances sub-spot resolution	yes (q)	Visium laminar/continuous tissue; want sub-spot enhancement	non-contiguous regions; MCMC is slow; smoothing strength is a fixed prior
BASS	Bayesian hierarchical: cell-type AND domain jointly, Potts prior, multi-sample	yes (C, D)	joint cell-type + domain, multi-sample, low-continuity tissue	heavier to run; needs both counts set
STAGATE	Graph attention autoencoder; learns per-edge weights, reconstructs from a spatially-smoothed latent	embedding free; downstream k for mclust	continuous tissue; attention down-weights cross-boundary edges (fights over-smoothing); scales to Slide-seq/Stereo-seq	black-box embedding; sensitive to graph radius
GraphST	Graph self-supervised contrastive learning	yes (mclust)	low-res Visium; also does integration + deconvolution	sensitive to graph construction
SpaGCN	GCN fusing expression + coordinates + histology RGB	searches resolution to hit target count	H&E histology is informative	needs registered histology
SEDR	Masked autoencoder + variational graph autoencoder	yes (mclust)	Visium/Slide-seq/Stereo-seq, robust on DLPFC	graph-construction sensitivity
BANKSY	Neighbor-AUGMENTED features: own + neighborhood-mean + azimuthal Gabor; then Leiden/k-means	no fixed k	high-res imaging AND sequencing; unifies cell typing (lambda0.2) and domains (lambda0.8); very scalable; transparent	lambda mis-set collapses the task it solves
UTAG	Message passing: multiply features by normalized adjacency (one-hop average), then Leiden	no fixed k	multiplexed imaging/proteomics (IMC, CODEX, MIBI); fast	one-hop smoothing only
stLearn (SME)	Histology-CNN-weighted smoothing of each spot's expression, then clustering	downstream Louvain/k-means	H&E available and well-registered	only as good as image registration

Mechanism families: MRF/Bayesian smoothing (BayesSpace, BASS, PRECAST) vs graph neural net (STAGATE, GraphST, SpaGCN, SEDR) vs neighbor-augmentation (BANKSY, UTAG) vs histology-guided (stLearn, SpaGCN). Benchmarks evolve fast -- verify the current verdict for the tissue geometry before committing.

Read the Benchmark With the DLPFC Caveat

The standard ground truth is DLPFC (Maynard 2021 Nat Neurosci 24:425-436): 12 Visium sections of human dorsolateral prefrontal cortex, manually annotated into 6 cortical layers + white matter, scored by ARI. The Yuan 2024 (Nat Methods 21:712-722) benchmark of 13 methods x 34 datasets concludes there is NO single winner -- methods are complementary across accuracy, spatial continuity, marker detection, scalability, and robustness. On DLPFC, GNN/graph methods (STAGATE, SEDR, DeepST) and BayesSpace are among the most robust, and a technology-stratified benchmark (Chen 2025 iMeta 4:e70084) finds STAGATE/GraphST best on low-resolution Visium while BASS/stLearn/BANKSY lead on high-resolution platforms.

The caveat: DLPFC is a CONTINUOUS, LAMINAR tissue, which flatters smoothing-friendly methods. Do not over-generalize these rankings to non-laminar tissue. ALL methods struggle with NON-CONTIGUOUS domains (the same region appearing in separated patches -- scattered tumor nests, immune aggregates) because the spatial prior assumes contiguity; for non-contiguous biology, lower the spatial weight or switch to a niche/neighborhood analysis rather than domain segmentation.

Build the Spatial Graph

Goal: Construct the spatial neighbor graph that every domain method inherits.

Approach: Use a hex lattice for Visium (6 neighbors) and a generic kNN/Delaunay graph for imaging point clouds; the graph radius IS a spatial-weight knob (too dense over-smooths, too sparse fragments).

import squidpy as sq
import scanpy as sc

adata = sc.read_h5ad('preprocessed.h5ad')

# Visium hex lattice: 6 immediate neighbors. For imaging point clouds use
# coord_type='generic' with KNN or Delaunay. Build in microns where possible --
# a graph built in pixels has a different radius than one built in microns.
sq.gr.spatial_neighbors(adata, coord_type='grid', n_neighs=6)

Expression-Only Clustering Is the Salt-and-Pepper Baseline

Goal: Show why a domain method is needed at all.

Approach: Cluster on expression PCA with no spatial term; the result is spatially incoherent and demonstrates the failure domain methods correct.

sc.pp.neighbors(adata, n_neighbors=15, n_pcs=30)
sc.tl.leiden(adata, resolution=0.5, key_added='expr_leiden',
             flavor='igraph', n_iterations=2, directed=False)

# Plot on tissue: speckled, no contiguous regions -- this is the baseline a
# domain method is built to beat, not a domain result.
sq.pl.spatial_scatter(adata, color='expr_leiden')

Neighbor-Augmented Domains (BANKSY-style) and the Spatial-Weight Knob

Goal: Produce spatially coherent domains and make the over-smoothing knob explicit.

Approach: BANKSY concatenates each cell's own expression with its neighborhood-mean expression, mixed by lambda; lambda0.2 yields cell typing, lambda0.8 yields domains. A transparent neighbor-augmented matrix reproduces the idea with Squidpy when the BANKSY package is unavailable -- the lambda here is the load-bearing decision, not the clustering call.

import numpy as np
from sklearn.preprocessing import normalize

# Mean expression over each spot's spatial neighbors (the neighborhood signal).
W = normalize(adata.obsp['spatial_connectivities'], norm='l1', axis=1)
neighbor_mean = W @ adata.obsm['X_pca']

# lambda is the spatial-weight knob: ~0.8 for domains, ~0.2 for cell typing.
# Too high -> blobs (boundaries erased); too low -> salt-and-pepper.
lam = 0.8
augmented = np.concatenate(
    [np.sqrt(1 - lam) * adata.obsm['X_pca'], np.sqrt(lam) * neighbor_mean], axis=1)
adata.obsm['X_banksy'] = augmented

sc.pp.neighbors(adata, use_rep='X_banksy', key_added='banksy')
sc.tl.leiden(adata, resolution=0.5, key_added='domains', neighbors_key='banksy',
             flavor='igraph', n_iterations=2, directed=False)
sq.pl.spatial_scatter(adata, color='domains')

If the BANKSY package is installed, prefer it: import banksy_py (Python) or the R Banksy Bioconductor package compute the own + neighborhood-mean + azimuthal Gabor (AGF) features and expose lambda directly.

Sweep the Spatial Weight to Find the Over-Smoothing Edge

Goal: Locate the lambda where boundaries are sharp but regions stay contiguous.

Approach: Recompute domains across a lambda grid and inspect boundaries against histology; the right lambda is the largest value before distinct regions merge into blobs, judged visually, not by an internal score.

for lam in [0.2, 0.5, 0.8]:
    aug = np.concatenate(
        [np.sqrt(1 - lam) * adata.obsm['X_pca'],
         np.sqrt(lam) * (W @ adata.obsm['X_pca'])], axis=1)
    adata.obsm['X_banksy'] = aug
    sc.pp.neighbors(adata, use_rep='X_banksy', key_added='banksy')
    sc.tl.leiden(adata, resolution=0.5, key_added=f'domains_l{lam}',
                 neighbors_key='banksy', flavor='igraph', n_iterations=2, directed=False)

# Low lambda -> speckled; high lambda -> over-smoothed blobs. Pick by eye
# against known tissue architecture, not by silhouette.
sq.pl.spatial_scatter(adata, color=['domains_l0.2', 'domains_l0.5', 'domains_l0.8'])

Choose k as a Biological Decision, With k+-1 Sensitivity

Goal: Set the number of domains to the regionalization the question needs and show the answer is not fragile to it.

Approach: When a method takes k directly (BayesSpace q, mclust on a GNN embedding), run k, k-1, k+1 and report all three; do not silently optimize k against a clustering score, which has no biological ground truth.

from sklearn.mixture import GaussianMixture

for k in [6, 7, 8]:  # DLPFC has 6 layers + white matter -> k near 7
    gm = GaussianMixture(n_components=k, covariance_type='full', random_state=0)
    adata.obs[f'domains_k{k}'] = gm.fit_predict(adata.obsm['X_banksy']).astype(str)

# Report k+-1 side by side; a domain that only appears at one k is a weak claim.
sq.pl.spatial_scatter(adata, color=['domains_k6', 'domains_k7', 'domains_k8'])

BayesSpace for Laminar Tissue (R)

Goal: Apply an MRF smoothing prior, the robust choice on continuous Visium tissue.

Approach: Preprocess, then spatialCluster with q domains; the smoothing prior couples neighboring spots. Run in R and import the labels.

library(BayesSpace)

sce <- readRDS('sce.rds')
sce <- spatialPreprocess(sce, platform = 'Visium', n.PCs = 15)
# q is the biological k; nrep is MCMC iterations. spatialEnhance() can split
# spots into subspots for higher resolution after spatialCluster().
sce <- spatialCluster(sce, q = 7, nrep = 10000)
write.csv(data.frame(barcode = colnames(sce), domain = sce$spatial.cluster),
          'bayesspace_domains.csv', row.names = FALSE)

STAGATE for High-Resolution or Large Sections (optional)

Goal: Learn a spatially-aware embedding whose attention down-weights cross-boundary edges.

Approach: Build the STAGATE radius graph, train the graph-attention autoencoder, then cluster the embedding; set the radius to the over-smoothing knob.

import STAGATE  # optional dependency; pip install STAGATE_pyG or use the TF build

STAGATE.Cal_Spatial_Net(adata, rad_cutoff=150)  # rad_cutoff is the graph radius knob
STAGATE.Stats_Spatial_Net(adata)
adata = STAGATE.train_STAGATE(adata)

sc.pp.neighbors(adata, use_rep='STAGATE')
sc.tl.leiden(adata, resolution=0.5, key_added='stagate_domains',
             flavor='igraph', n_iterations=2, directed=False)

Name Domains From Markers

Goal: Attach anatomical labels to spatially coherent clusters.

Approach: Rank per-domain markers, then map cluster IDs to region names; marker p-values from the same data that defined the domains are for ranking and labeling only, not inference (the double-dipping caveat from single-cell/clustering).

sc.tl.rank_genes_groups(adata, groupby='domains', method='wilcoxon')
markers = sc.get.rank_genes_groups_df(adata, group=None)
print(markers.groupby('group').head(5))

domain_names = {'0': 'White matter', '1': 'Layer 1', '2': 'Layer 2/3'}
adata.obs['region'] = adata.obs['domains'].map(domain_names)
sq.pl.spatial_scatter(adata, color='region')

Common Errors

Symptom	Cause	Fix
Speckled, spatially incoherent labels	No spatial term (plain Leiden on expression) or spatial weight too low	Use a domain method; raise lambda / smoothing / graph density
Regions merged into smooth blobs, boundaries gone	Over-smoothing -- spatial weight too high or graph radius too large	Lower lambda / smoothing strength / rad_cutoff; check boundaries against histology
Scattered tumor nests collapse into one domain or vanish	Non-contiguous domain; spatial prior assumes contiguity (Yuan 2024)	Lower the spatial weight, or switch to niche/neighborhood analysis (spatial-statistics)
Domain count feels arbitrary / reviewer questions k	k optimized against a silhouette score instead of chosen biologically	Set k from the biology; report k+-1 sensitivity; justify the regionalization
Domains track a single sample/section	Batch confounded with biology in multi-sample data	Use a multi-sample method (BASS, PRECAST, GraphST integration) or integrate first
Answer changes with no parameter change	Spatial graph rebuilt with different units (pixels vs microns) or different kNN/Delaunay	Pin the graph construction and coordinate units; build once, reuse
"Domain" answer to a "which types co-occur" question	Domain segmentation used for a niche question	Use neighborhood enrichment / co-occurrence (spatial-statistics) instead
Marker p-values quoted as proof a domain is real	Double-dipping (testing the clustering that defined the groups)	Use markers for ranking/labeling only; validate regions against known architecture

Related Skills

• spatial-neighbors - Build and tune the spatial graph every domain method inherits
• spatial-statistics - Niche/neighborhood enrichment and co-occurrence when the question is which cell types co-occur, not which region this is
• spatial-deconvolution - Per-spot cell-type composition; domains are regions, deconvolution is composition within a spot
• single-cell/clustering - Non-spatial clustering, resolution sweeps, and the double-dipping caveat on post-clustering marker tests

References

• Zhao et al. (2021). Spatial transcriptomics at subspot resolution with BayesSpace. Nat Biotechnol 39:1375-1384.
• Hu et al. (2021). SpaGCN: integrating gene expression, spatial location and histology to identify spatial domains and SVGs by graph convolutional network. Nat Methods 18:1342-1351.
• Dong & Zhang (2022). Deciphering spatial domains from spatially resolved transcriptomics with an adaptive graph attention auto-encoder (STAGATE). Nat Commun 13:1739.
• Long et al. (2023). Spatially informed clustering, integration, and deconvolution of spatial transcriptomics with GraphST. Nat Commun 14:1155.
• Singhal et al. (2024). BANKSY unifies cell typing and tissue domain segmentation for scalable spatial omics data analysis. Nat Genet 56:431-441.
• Kim et al. (2022). Unsupervised discovery of tissue architecture in multiplexed imaging (UTAG). Nat Methods 19:1653-1661.
• Xu et al. (2024). Unsupervised spatially embedded deep representation of spatial transcriptomics (SEDR). Genome Med 16:12.
• Li & Zhou (2022). BASS: multi-scale and multi-sample analysis enables accurate cell type clustering and spatial domain detection in spatial transcriptomic studies. Genome Biol 23:168.
• Yuan et al. (2024). Benchmarking spatial clustering methods with spatially resolved transcriptomics data. Nat Methods 21:712-722.
• Chen et al. (2025). A comprehensive benchmarking for spatially resolved transcriptomics clustering methods across variable technologies, organs, and replicates. iMeta 4:e70084.
• Maynard et al. (2021). Transcriptome-scale spatial gene expression in the human dorsolateral prefrontal cortex. Nat Neurosci 24:425-436.
• Palla et al. (2022). Squidpy: a scalable framework for spatial omics analysis. Nat Methods 19:171-178.