Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916838541099008 |
|---|---|
| author | Liang, Yupu Zhang, Yaping Zhang, Zhiyang Chen, Zhiyuan Zhao, Yang Xiang, Lu Zong, Chengqing Zhou, Yu |
| author_facet | Liang, Yupu Zhang, Yaping Zhang, Zhiyang Chen, Zhiyuan Zhao, Yang Xiang, Lu Zong, Chengqing Zhou, Yu |
| contents | Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_08309 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency Liang, Yupu Zhang, Yaping Zhang, Zhiyang Chen, Zhiyuan Zhao, Yang Xiang, Lu Zong, Chengqing Zhou, Yu Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) have shown strong performance in document image tasks, especially Optical Character Recognition (OCR). However, they struggle with Document Image Machine Translation (DIMT), which requires handling both cross-modal and cross-lingual challenges. Previous efforts to enhance DIMT capability through Supervised Fine-Tuning (SFT) on the DIMT dataset often result in the forgetting of the model's existing monolingual abilities, such as OCR. To address these challenges, we introduce a novel fine-tuning paradigm, named Synchronously Self-Reviewing (SSR) its OCR proficiency, inspired by the concept "Bilingual Cognitive Advantage". Specifically, SSR prompts the model to generate OCR text before producing translation text, which allows the model to leverage its strong monolingual OCR ability while learning to translate text across languages. Comprehensive experiments demonstrate the proposed SSR learning helps mitigate catastrophic forgetting, improving the generalization ability of MLLMs on both OCR and DIMT tasks. |
| title | Improving MLLM's Document Image Machine Translation via Synchronously Self-reviewing Its OCR Proficiency |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2507.08309 |