Nichevi: A Probabilistic Framework to Embed Cellular Interaction in Spatial Transcriptomics

Abstract

Spatial transcriptomics has the potential to reveal cellular interactions by measuring gene expression in situ while maintaining the tissue context of each cell. Existing deep learning methods for non-spatial single-cell omics optimize cellular embeddings of gene expression. They enable the harmonization between experimental batches while embedding the variation of the cell state. Spatial transcrip- tomics allows one to study the cell state composition of a spatial neighborhood. These cellular niches confine the tissue organization and encompass functional units of an organ. However, computational methods for encoding meaningful low- dimensional representations of both gene expression and cell states of neighboring cells a are currently lacking. Here, we introduce NicheVI, a deep learning model that decodes gene expression, niche cell-type composition, and variation in cell state of other cells within a niche. In case studies, NicheVI uncovered additional fine-grained heterogeneity of cell-types not captured by non-spatial and other spatially aware models and corresponding to the cellular niche