MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916623572533248 |
|---|---|
| author | Tang, Haoran Cao, Meng Huang, Jinfa Liu, Ruyang Jin, Peng Li, Ge Liang, Xiaodan |
| author_facet | Tang, Haoran Cao, Meng Huang, Jinfa Liu, Ruyang Jin, Peng Li, Ge Liang, Xiaodan |
| contents | Text-Video Retrieval (TVR) aims to align and associate relevant video content with corresponding natural language queries. Most existing TVR methods are based on large-scale pre-trained vision-language models (e.g., CLIP). However, due to the inherent plain structure of CLIP, few TVR methods explore the multi-scale representations which offer richer contextual information for a more thorough understanding. To this end, we propose MUSE, a multi-scale mamba with linear computational complexity for efficient cross-resolution modeling. Specifically, the multi-scale representations are generated by applying a feature pyramid on the last single-scale feature map. Then, we employ the Mamba structure as an efficient multi-scale learner to jointly learn scale-wise representations. Furthermore, we conduct comprehensive studies to investigate different model structures and designs. Extensive results on three popular benchmarks have validated the superiority of MUSE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10575 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval Tang, Haoran Cao, Meng Huang, Jinfa Liu, Ruyang Jin, Peng Li, Ge Liang, Xiaodan Computer Vision and Pattern Recognition Text-Video Retrieval (TVR) aims to align and associate relevant video content with corresponding natural language queries. Most existing TVR methods are based on large-scale pre-trained vision-language models (e.g., CLIP). However, due to the inherent plain structure of CLIP, few TVR methods explore the multi-scale representations which offer richer contextual information for a more thorough understanding. To this end, we propose MUSE, a multi-scale mamba with linear computational complexity for efficient cross-resolution modeling. Specifically, the multi-scale representations are generated by applying a feature pyramid on the last single-scale feature map. Then, we employ the Mamba structure as an efficient multi-scale learner to jointly learn scale-wise representations. Furthermore, we conduct comprehensive studies to investigate different model structures and designs. Extensive results on three popular benchmarks have validated the superiority of MUSE. |
| title | MUSE: Mamba is Efficient Multi-scale Learner for Text-video Retrieval |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.10575 |