Enhance the Robustness of Text-Centric Multimodal Alignments
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929411631087616 |
|---|---|
| author | Yen, Ting-Yu Tsai, Yun-Da Liao, Keng-Te Lin, Shou-De |
| author_facet | Yen, Ting-Yu Tsai, Yun-Da Liao, Keng-Te Lin, Shou-De |
| contents | Converting different modalities into general text, serving as input prompts for large language models (LLMs), is a common method to align multimodal models when there is limited pairwise data. This text-centric approach leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation. This enables downstream models to effectively interpret various modal inputs. This study assesses the quality and robustness of multimodal representations in the presence of missing entries, noise, or absent modalities, revealing that current text-centric alignment methods compromise downstream robustness. To address this issue, we propose a new text-centric approach that achieves superior robustness compared to previous methods across various modalities in different settings. Our findings highlight the potential of this approach to enhance the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_05036 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhance the Robustness of Text-Centric Multimodal Alignments Yen, Ting-Yu Tsai, Yun-Da Liao, Keng-Te Lin, Shou-De Computation and Language Machine Learning Converting different modalities into general text, serving as input prompts for large language models (LLMs), is a common method to align multimodal models when there is limited pairwise data. This text-centric approach leverages the unique properties of text as a modality space, transforming diverse inputs into a unified textual representation. This enables downstream models to effectively interpret various modal inputs. This study assesses the quality and robustness of multimodal representations in the presence of missing entries, noise, or absent modalities, revealing that current text-centric alignment methods compromise downstream robustness. To address this issue, we propose a new text-centric approach that achieves superior robustness compared to previous methods across various modalities in different settings. Our findings highlight the potential of this approach to enhance the robustness and adaptability of multimodal representations, offering a promising solution for dynamic and real-world applications. |
| title | Enhance the Robustness of Text-Centric Multimodal Alignments |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2407.05036 |