Tell2Reg: Establishing spatial correspondence between images by the same language prompts
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910815165087744 |
|---|---|
| author | Yan, Wen Yang, Qianye Huang, Shiqi Wang, Yipei Punwani, Shonit Emberton, Mark Stavrinides, Vasilis Hu, Yipeng Barratt, Dean |
| author_facet | Yan, Wen Yang, Qianye Huang, Shiqi Wang, Yipei Punwani, Shonit Emberton, Mark Stavrinides, Vasilis Hu, Yipeng Barratt, Dean |
| contents | Spatial correspondence can be represented by pairs of segmented regions, such that the image registration networks aim to segment corresponding regions rather than predicting displacement fields or transformation parameters. In this work, we show that such a corresponding region pair can be predicted by the same language prompt on two different images using the pre-trained large multimodal models based on GroundingDINO and SAM. This enables a fully automated and training-free registration algorithm, potentially generalisable to a wide range of image registration tasks. In this paper, we present experimental results using one of the challenging tasks, registering inter-subject prostate MR images, which involves both highly variable intensity and morphology between patients. Tell2Reg is training-free, eliminating the need for costly and time-consuming data curation and labelling that was previously required for this registration task. This approach outperforms unsupervised learning-based registration methods tested, and has a performance comparable to weakly-supervised methods. Additional qualitative results are also presented to suggest that, for the first time, there is a potential correlation between language semantics and spatial correspondence, including the spatial invariance in language-prompted regions and the difference in language prompts between the obtained local and global correspondences. Code is available at https://github.com/yanwenCi/Tell2Reg.git. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03118 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tell2Reg: Establishing spatial correspondence between images by the same language prompts Yan, Wen Yang, Qianye Huang, Shiqi Wang, Yipei Punwani, Shonit Emberton, Mark Stavrinides, Vasilis Hu, Yipeng Barratt, Dean Computer Vision and Pattern Recognition Artificial Intelligence Image and Video Processing 00B25 I.2.7 Spatial correspondence can be represented by pairs of segmented regions, such that the image registration networks aim to segment corresponding regions rather than predicting displacement fields or transformation parameters. In this work, we show that such a corresponding region pair can be predicted by the same language prompt on two different images using the pre-trained large multimodal models based on GroundingDINO and SAM. This enables a fully automated and training-free registration algorithm, potentially generalisable to a wide range of image registration tasks. In this paper, we present experimental results using one of the challenging tasks, registering inter-subject prostate MR images, which involves both highly variable intensity and morphology between patients. Tell2Reg is training-free, eliminating the need for costly and time-consuming data curation and labelling that was previously required for this registration task. This approach outperforms unsupervised learning-based registration methods tested, and has a performance comparable to weakly-supervised methods. Additional qualitative results are also presented to suggest that, for the first time, there is a potential correlation between language semantics and spatial correspondence, including the spatial invariance in language-prompted regions and the difference in language prompts between the obtained local and global correspondences. Code is available at https://github.com/yanwenCi/Tell2Reg.git. |
| title | Tell2Reg: Establishing spatial correspondence between images by the same language prompts |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Image and Video Processing 00B25 I.2.7 |
| url | https://arxiv.org/abs/2502.03118 |