Diagram Formalization Enhanced Multi-Modal Geometry Problem Solver
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910594713518080 |
|---|---|
| author | Zhang, Zeren Cheng, Jo-Ku Deng, Jingyang Tian, Lu Ma, Jinwen Qin, Ziran Zhang, Xiaokai Zhu, Na Leng, Tuo |
| author_facet | Zhang, Zeren Cheng, Jo-Ku Deng, Jingyang Tian, Lu Ma, Jinwen Qin, Ziran Zhang, Xiaokai Zhu, Na Leng, Tuo |
| contents | Mathematical reasoning remains an ongoing challenge for AI models, especially for geometry problems that require both linguistic and visual signals. As the vision encoders of most MLLMs are trained on natural scenes, they often struggle to understand geometric diagrams, performing no better in geometry problem solving than LLMs that only process text. This limitation is amplified by the lack of effective methods for representing geometric relationships. To address these issues, we introduce the Diagram Formalization Enhanced Geometry Problem Solver (DFE-GPS), a new framework that integrates visual features, geometric formal language, and natural language representations. We propose a novel synthetic data approach and create a large-scale geometric dataset, SynthGeo228K, annotated with both formal and natural language captions, designed to enhance the vision encoder for a better understanding of geometric structures. Our framework improves MLLMs' ability to process geometric diagrams and extends their application to open-ended tasks on the formalgeo7k dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_04214 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diagram Formalization Enhanced Multi-Modal Geometry Problem Solver Zhang, Zeren Cheng, Jo-Ku Deng, Jingyang Tian, Lu Ma, Jinwen Qin, Ziran Zhang, Xiaokai Zhu, Na Leng, Tuo Computer Vision and Pattern Recognition Mathematical reasoning remains an ongoing challenge for AI models, especially for geometry problems that require both linguistic and visual signals. As the vision encoders of most MLLMs are trained on natural scenes, they often struggle to understand geometric diagrams, performing no better in geometry problem solving than LLMs that only process text. This limitation is amplified by the lack of effective methods for representing geometric relationships. To address these issues, we introduce the Diagram Formalization Enhanced Geometry Problem Solver (DFE-GPS), a new framework that integrates visual features, geometric formal language, and natural language representations. We propose a novel synthetic data approach and create a large-scale geometric dataset, SynthGeo228K, annotated with both formal and natural language captions, designed to enhance the vision encoder for a better understanding of geometric structures. Our framework improves MLLMs' ability to process geometric diagrams and extends their application to open-ended tasks on the formalgeo7k dataset. |
| title | Diagram Formalization Enhanced Multi-Modal Geometry Problem Solver |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.04214 |