SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918249460924416 |
|---|---|
| author | Jeong, Dabin Vahidi, Amirhossein Ramírez-Suástegui, Ciro Moullet, Marie Ly, Kevin Sanian, Mohammad Vali Birk, Sebastian Chang, Yinshui Boxall, Adam Jafree, Daniyal Steele, Lloyd MS, Vijaya Baskar Haniffa, Muzlifah Lotfollahi, Mohammad |
| author_facet | Jeong, Dabin Vahidi, Amirhossein Ramírez-Suástegui, Ciro Moullet, Marie Ly, Kevin Sanian, Mohammad Vali Birk, Sebastian Chang, Yinshui Boxall, Adam Jafree, Daniyal Steele, Lloyd MS, Vijaya Baskar Haniffa, Muzlifah Lotfollahi, Mohammad |
| contents | Recent advances in computational pathology have leveraged vision-language models to learn joint representations of Hematoxylin and Eosin (HE) images with spatial transcriptomic (ST) profiles. However, existing approaches typically align HE tiles with their corresponding ST profiles at a single scale, overlooking fine-grained cellular structures and their spatial organization. To address this, we propose Sigmma, a multi-modal contrastive alignment framework for learning hierarchical representations of HE images and spatial transcriptome profiles across multiple scales. Sigmma introduces multi-scale contrastive alignment, ensuring that representations learned at different scales remain coherent across modalities. Furthermore, by representing cell interactions as a graph and integrating inter- and intra-subgraph relationships, our approach effectively captures cell-cell interactions, ranging from fine to coarse, within the tissue microenvironment. We demonstrate that Sigmm learns representations that better capture cross-modal correspondences, leading to an improvement of avg. 9.78\% in the gene-expression prediction task and avg. 26.93\% in the cross-modal retrieval task across datasets. We further show that it learns meaningful multi-tissue organization in downstream analyses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_15464 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome Jeong, Dabin Vahidi, Amirhossein Ramírez-Suástegui, Ciro Moullet, Marie Ly, Kevin Sanian, Mohammad Vali Birk, Sebastian Chang, Yinshui Boxall, Adam Jafree, Daniyal Steele, Lloyd MS, Vijaya Baskar Haniffa, Muzlifah Lotfollahi, Mohammad Computer Vision and Pattern Recognition Machine Learning Recent advances in computational pathology have leveraged vision-language models to learn joint representations of Hematoxylin and Eosin (HE) images with spatial transcriptomic (ST) profiles. However, existing approaches typically align HE tiles with their corresponding ST profiles at a single scale, overlooking fine-grained cellular structures and their spatial organization. To address this, we propose Sigmma, a multi-modal contrastive alignment framework for learning hierarchical representations of HE images and spatial transcriptome profiles across multiple scales. Sigmma introduces multi-scale contrastive alignment, ensuring that representations learned at different scales remain coherent across modalities. Furthermore, by representing cell interactions as a graph and integrating inter- and intra-subgraph relationships, our approach effectively captures cell-cell interactions, ranging from fine to coarse, within the tissue microenvironment. We demonstrate that Sigmm learns representations that better capture cross-modal correspondences, leading to an improvement of avg. 9.78\% in the gene-expression prediction task and avg. 26.93\% in the cross-modal retrieval task across datasets. We further show that it learns meaningful multi-tissue organization in downstream analyses. |
| title | SIGMMA: Hierarchical Graph-Based Multi-Scale Multi-modal Contrastive Alignment of Histopathology Image and Spatial Transcriptome |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2511.15464 |