Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913936008282112 |
|---|---|
| author | Liang, Yupu Zhang, Yaping Zhang, Zhiyang Zhao, Yang Xiang, Lu Zong, Chengqing Zhou, Yu |
| author_facet | Liang, Yupu Zhang, Yaping Zhang, Zhiyang Zhao, Yang Xiang, Lu Zong, Chengqing Zhou, Yu |
| contents | Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07572 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation Liang, Yupu Zhang, Yaping Zhang, Zhiyang Zhao, Yang Xiang, Lu Zong, Chengqing Zhou, Yu Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Document Image Machine Translation (DIMT) aims to translate text within document images, facing generalization challenges due to limited training data and the complex interplay between visual and textual information. To address these challenges, we introduce M4Doc, a novel single-to-mix modality alignment framework leveraging Multimodal Large Language Models (MLLMs). M4Doc aligns an image-only encoder with the multimodal representations of an MLLM, pre-trained on large-scale document image datasets. This alignment enables a lightweight DIMT model to learn crucial visual-textual correlations during training. During inference, M4Doc bypasses the MLLM, maintaining computational efficiency while benefiting from its multimodal knowledge. Comprehensive experiments demonstrate substantial improvements in translation quality, especially in cross-domain generalization and challenging document image scenarios. |
| title | Single-to-mix Modality Alignment with Multimodal Large Language Model for Document Image Machine Translation |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.07572 |