Zapisane w:
| Główni autorzy: | , , , , |
|---|---|
| Format: | Preprint |
| Wydane: |
2024
|
| Hasła przedmiotowe: | |
| Dostęp online: | https://arxiv.org/abs/2410.08021 |
| Etykiety: |
Dodaj etykietę
Nie ma etykietki, Dołącz pierwszą etykiete!
|
| _version_ | 1866914987940773888 |
|---|---|
| author | Xiao, Linhui Yang, Xiaoshan Peng, Fang Wang, Yaowei Xu, Changsheng |
| author_facet | Xiao, Linhui Yang, Xiaoshan Peng, Fang Wang, Yaowei Xu, Changsheng |
| contents | Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08021 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling Xiao, Linhui Yang, Xiaoshan Peng, Fang Wang, Yaowei Xu, Changsheng Computer Vision and Pattern Recognition Constrained by the separate encoding of vision and language, existing grounding and referring segmentation works heavily rely on bulky Transformer-based fusion en-/decoders and a variety of early-stage interaction technologies. Simultaneously, the current mask visual language modeling (MVLM) fails to capture the nuanced referential relationship between image-text in referring tasks. In this paper, we propose OneRef, a minimalist referring framework built on the modality-shared one-tower transformer that unifies the visual and linguistic feature spaces. To modeling the referential relationship, we introduce a novel MVLM paradigm called Mask Referring Modeling (MRefM), which encompasses both referring-aware mask image modeling and referring-aware mask language modeling. Both modules not only reconstruct modality-related content but also cross-modal referring content. Within MRefM, we propose a referring-aware dynamic image masking strategy that is aware of the referred region rather than relying on fixed ratios or generic random masking schemes. By leveraging the unified visual language feature space and incorporating MRefM's ability to model the referential relations, our approach enables direct regression of the referring results without resorting to various complex techniques. Our method consistently surpasses existing approaches and achieves SoTA performance on both grounding and segmentation tasks, providing valuable insights for future research. Our code and models are available at https://github.com/linhuixiao/OneRef. |
| title | OneRef: Unified One-tower Expression Grounding and Segmentation with Mask Referring Modeling |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.08021 |