How to Utilize Complementary Vision-Text Information for 2D Structure Understanding
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908893275226112 |
|---|---|
| author | Dong, Jiancheng Jia, Pengyue Xu, Derong Cheng, Jiawei Peng, Jingyu Zhang, Chao Liu, Bowen Sun, Xin Su, Lixin Wang, Shuaiqiang Yin, Dawei Zhao, Xiangyu |
| author_facet | Dong, Jiancheng Jia, Pengyue Xu, Derong Cheng, Jiawei Peng, Jingyu Zhang, Chao Liu, Bowen Sun, Xin Su, Lixin Wang, Shuaiqiang Yin, Dawei Zhao, Xiangyu |
| contents | LLMs typically linearize 2D tables into 1D sequences to fit their autoregressive architecture, which weakens row-column adjacency and other layout cues. In contrast, purely visual encoders can capture spatial cues, yet often struggle to preserve exact cell text. Our analysis reveals that these two modalities provide highly distinct information to LLMs and exhibit strong complementarity. However, direct concatenation and other fusion methods yield limited gains and frequently introduce cross-modal interference. To address this issue, we propose DiVA-Former, a lightweight architecture designed to effectively integrate vision and text information. DiVA-Former leverages visual tokens as dynamic queries to distill long textual sequences into digest vectors, thereby effectively exploiting complementary vision--text information. Evaluated across 13 table benchmarks, DiVA-Former improves upon the pure-text baseline by 23.9\% and achieves consistent gains over existing baselines using visual inputs, textual inputs, or a combination of both. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_16245 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How to Utilize Complementary Vision-Text Information for 2D Structure Understanding Dong, Jiancheng Jia, Pengyue Xu, Derong Cheng, Jiawei Peng, Jingyu Zhang, Chao Liu, Bowen Sun, Xin Su, Lixin Wang, Shuaiqiang Yin, Dawei Zhao, Xiangyu Computer Vision and Pattern Recognition Computation and Language LLMs typically linearize 2D tables into 1D sequences to fit their autoregressive architecture, which weakens row-column adjacency and other layout cues. In contrast, purely visual encoders can capture spatial cues, yet often struggle to preserve exact cell text. Our analysis reveals that these two modalities provide highly distinct information to LLMs and exhibit strong complementarity. However, direct concatenation and other fusion methods yield limited gains and frequently introduce cross-modal interference. To address this issue, we propose DiVA-Former, a lightweight architecture designed to effectively integrate vision and text information. DiVA-Former leverages visual tokens as dynamic queries to distill long textual sequences into digest vectors, thereby effectively exploiting complementary vision--text information. Evaluated across 13 table benchmarks, DiVA-Former improves upon the pure-text baseline by 23.9\% and achieves consistent gains over existing baselines using visual inputs, textual inputs, or a combination of both. |
| title | How to Utilize Complementary Vision-Text Information for 2D Structure Understanding |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2603.16245 |