HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914330145980416 |
|---|---|
| author | Wang, Haolan Liu, Zhenghao Li, Xinze Yang, Xiaocui Gu, Yu Yan, Yukun Shi, Qi Li, Fangfang Chen, Chong Yu, Ge |
| author_facet | Wang, Haolan Liu, Zhenghao Li, Xinze Yang, Xiaocui Gu, Yu Yan, Yukun Shi, Qi Li, Fangfang Chen, Chong Yu, Ge |
| contents | Tabular data contains rich structural semantics and plays a crucial role in organizing and manipulating information. Recent methods employ Multi-modal Large Language Models (MLLMs) to address table-related tasks across various modalities of table representations. However, existing studies mainly focus on exploring the table understanding ability of MLLMs using unimodal representations, which limits further exploration of multi-modal representations to enable more effective table reasoning. To better capture structural semantics from the tabular data, this paper introduces the HybrId-modal Preference oPtimizatiOn (HIPPO) model, which represents tables using both text and image, optimizing MLLMs by learning more comprehensive table information from these multiple modalities. Specifically, HIPPO samples MLLM responses from hybrid-modal table representations and designs a modality-consistent sampling strategy to enhance response diversity and mitigate modality bias during Direct Preference Optimization (DPO) training. Experiments on table question answering and table fact verification tasks demonstrate the effectiveness of HIPPO, achieving a 4% improvement over various table reasoning models. Further analysis reveals that HIPPO not only enhances the table reasoning capability based on unimodal representations but also facilitates the extraction of complementary semantics across modalities. The code is available at https://github.com/NEUIR/HIPPO. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17315 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization Wang, Haolan Liu, Zhenghao Li, Xinze Yang, Xiaocui Gu, Yu Yan, Yukun Shi, Qi Li, Fangfang Chen, Chong Yu, Ge Computation and Language Tabular data contains rich structural semantics and plays a crucial role in organizing and manipulating information. Recent methods employ Multi-modal Large Language Models (MLLMs) to address table-related tasks across various modalities of table representations. However, existing studies mainly focus on exploring the table understanding ability of MLLMs using unimodal representations, which limits further exploration of multi-modal representations to enable more effective table reasoning. To better capture structural semantics from the tabular data, this paper introduces the HybrId-modal Preference oPtimizatiOn (HIPPO) model, which represents tables using both text and image, optimizing MLLMs by learning more comprehensive table information from these multiple modalities. Specifically, HIPPO samples MLLM responses from hybrid-modal table representations and designs a modality-consistent sampling strategy to enhance response diversity and mitigate modality bias during Direct Preference Optimization (DPO) training. Experiments on table question answering and table fact verification tasks demonstrate the effectiveness of HIPPO, achieving a 4% improvement over various table reasoning models. Further analysis reveals that HIPPO not only enhances the table reasoning capability based on unimodal representations but also facilitates the extraction of complementary semantics across modalities. The code is available at https://github.com/NEUIR/HIPPO. |
| title | HIPPO: Enhancing the Table Understanding Capability of LLMs through Hybrid-Modal Preference Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.17315 |