Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.02671 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913775455567872 |
|---|---|
| author | Liu, Zishuo Villarreal, Carlos Rabat Rahgouy, Mostafa Das, Amit Zhang, Zheng Ren, Chang Feng, Dongji |
| author_facet | Liu, Zishuo Villarreal, Carlos Rabat Rahgouy, Mostafa Das, Amit Zhang, Zheng Ren, Chang Feng, Dongji |
| contents | Fermi Problems (FPs) are mathematical reasoning tasks that require human-like logic and numerical reasoning. Unlike other reasoning questions, FPs often involve real-world impracticalities or ambiguous concepts, making them challenging even for humans to solve. Despite advancements in AI, particularly with large language models (LLMs) in various reasoning tasks, FPs remain relatively under-explored. This work conducted an exploratory study to examine the capabilities and limitations of LLMs in solving FPs. We first evaluated the overall performance of three advanced LLMs using a publicly available FP dataset. We designed prompts according to the recently proposed TELeR taxonomy, including a zero-shot scenario. Results indicated that all three LLMs achieved a fp_score (range between 0 - 1) below 0.5, underscoring the inherent difficulty of these reasoning tasks. To further investigate, we categorized FPs into standard and specific questions, hypothesizing that LLMs would perform better on standard questions, which are characterized by clarity and conciseness, than on specific ones. Comparative experiments confirmed this hypothesis, demonstrating that LLMs performed better on standard FPs in terms of both accuracy and efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02671 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLM for Complex Reasoning Task: An Exploratory Study in Fermi Problems Liu, Zishuo Villarreal, Carlos Rabat Rahgouy, Mostafa Das, Amit Zhang, Zheng Ren, Chang Feng, Dongji Computation and Language Fermi Problems (FPs) are mathematical reasoning tasks that require human-like logic and numerical reasoning. Unlike other reasoning questions, FPs often involve real-world impracticalities or ambiguous concepts, making them challenging even for humans to solve. Despite advancements in AI, particularly with large language models (LLMs) in various reasoning tasks, FPs remain relatively under-explored. This work conducted an exploratory study to examine the capabilities and limitations of LLMs in solving FPs. We first evaluated the overall performance of three advanced LLMs using a publicly available FP dataset. We designed prompts according to the recently proposed TELeR taxonomy, including a zero-shot scenario. Results indicated that all three LLMs achieved a fp_score (range between 0 - 1) below 0.5, underscoring the inherent difficulty of these reasoning tasks. To further investigate, we categorized FPs into standard and specific questions, hypothesizing that LLMs would perform better on standard questions, which are characterized by clarity and conciseness, than on specific ones. Comparative experiments confirmed this hypothesis, demonstrating that LLMs performed better on standard FPs in terms of both accuracy and efficiency. |
| title | LLM for Complex Reasoning Task: An Exploratory Study in Fermi Problems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.02671 |