Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915637771632640 |
|---|---|
| author | Truong, Thanh-Dat Tran, Huu-Thien Son, Tran Thai Raj, Bhiksha Luu, Khoa |
| author_facet | Truong, Thanh-Dat Tran, Huu-Thien Son, Tran Thai Raj, Bhiksha Luu, Khoa |
| contents | Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability in various understanding tasks. However, these models still suffer from some fundamental limitations related to robustness and generalization due to the alignment and correlation between visual and textual features. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding, and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM's pre-training and fine-tuning phases. In addition, we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to reconstruct the correct order of visual inputs. Then, we introduce a new Image-to-Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs on academic task-oriented and instruction-following LMM benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_14264 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models Truong, Thanh-Dat Tran, Huu-Thien Son, Tran Thai Raj, Bhiksha Luu, Khoa Computer Vision and Pattern Recognition Large multimodal models (LMMs) have gained impressive performance due to their outstanding capability in various understanding tasks. However, these models still suffer from some fundamental limitations related to robustness and generalization due to the alignment and correlation between visual and textual features. In this paper, we introduce a simple but efficient learning mechanism for improving the robust alignment between visual and textual modalities by solving shuffling problems. In particular, the proposed approach can improve reasoning capability, visual understanding, and cross-modality alignment by introducing two new tasks: reconstructing the image order and the text order into the LMM's pre-training and fine-tuning phases. In addition, we propose a new directed-token approach to capture visual and textual knowledge, enabling the capability to reconstruct the correct order of visual inputs. Then, we introduce a new Image-to-Response Guided loss to further improve the visual understanding of the LMM in its responses. The proposed approach consistently achieves state-of-the-art (SoTA) performance compared with prior LMMs on academic task-oriented and instruction-following LMM benchmarks. |
| title | Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2508.14264 |