Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.12692 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918269930176512 |
|---|---|
| author | Li, Xiao Zhu, Bolin Shi, Kaiwen Liu, Sichen Zhu, Yin Liu, Yiwei Cheng, Gong |
| author_facet | Li, Xiao Zhu, Bolin Shi, Kaiwen Liu, Sichen Zhu, Yin Liu, Yiwei Cheng, Gong |
| contents | The application of physics formulas is a fundamental human capability in numerical reasoning. While existing datasets often rely on implicit mathematical knowledge, they rarely explicitate the underlying formulas. To address this, we introduce FormulaReasoning, a new benchmark for formula-based numerical reasoning comprising 5,324 questions requiring calculations grounded in external physics principles. We provide high-quality, fine-grained annotations in English and Chinese--including formula structures, parameter names, symbols, values, and units--curated through manual effort and LLM-assisted validation. Additionally, we provide a consolidated formula database as an external knowledge source. To further challenge model performance, we develop an extended version of the dataset by coupling multiple questions. We evaluate various architectural and methodological frameworks, including retrieval-augmented methods, modular reasoning (formula generation, parameter extraction, and calculation), and preference-based optimization. Our analysis identifies critical challenges in formula-based reasoning, highlighting significant opportunities for future methodological advancement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_12692 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FormulaReasoning: A Dataset for Formula-Based Numerical Reasoning Li, Xiao Zhu, Bolin Shi, Kaiwen Liu, Sichen Zhu, Yin Liu, Yiwei Cheng, Gong Computation and Language The application of physics formulas is a fundamental human capability in numerical reasoning. While existing datasets often rely on implicit mathematical knowledge, they rarely explicitate the underlying formulas. To address this, we introduce FormulaReasoning, a new benchmark for formula-based numerical reasoning comprising 5,324 questions requiring calculations grounded in external physics principles. We provide high-quality, fine-grained annotations in English and Chinese--including formula structures, parameter names, symbols, values, and units--curated through manual effort and LLM-assisted validation. Additionally, we provide a consolidated formula database as an external knowledge source. To further challenge model performance, we develop an extended version of the dataset by coupling multiple questions. We evaluate various architectural and methodological frameworks, including retrieval-augmented methods, modular reasoning (formula generation, parameter extraction, and calculation), and preference-based optimization. Our analysis identifies critical challenges in formula-based reasoning, highlighting significant opportunities for future methodological advancement. |
| title | FormulaReasoning: A Dataset for Formula-Based Numerical Reasoning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.12692 |