Evaluating LLM Reasoning in the Operations Research Domain with ORQA
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913684461191168 |
|---|---|
| author | Mostajabdaveh, Mahdi Yu, Timothy T. Dash, Samarendra Chandan Bindu Ramamonjison, Rindranirina Byusa, Jabo Serge Carenini, Giuseppe Zhou, Zirui Zhang, Yong |
| author_facet | Mostajabdaveh, Mahdi Yu, Timothy T. Dash, Samarendra Chandan Bindu Ramamonjison, Rindranirina Byusa, Jabo Serge Carenini, Giuseppe Zhou, Zirui Zhang, Yong |
| contents | In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark designed to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark evaluates whether LLMs can emulate the knowledge and reasoning skills of OR experts when confronted with diverse and complex optimization problems. The dataset, developed by OR experts, features real-world optimization problems that demand multistep reasoning to construct their mathematical models. Our evaluations of various open source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral, reveal their modest performance, highlighting a gap in their ability to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs generalization capabilities, offering valuable insights for future research in this area. The dataset and evaluation code are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_17874 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating LLM Reasoning in the Operations Research Domain with ORQA Mostajabdaveh, Mahdi Yu, Timothy T. Dash, Samarendra Chandan Bindu Ramamonjison, Rindranirina Byusa, Jabo Serge Carenini, Giuseppe Zhou, Zirui Zhang, Yong Computation and Language Artificial Intelligence In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark designed to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark evaluates whether LLMs can emulate the knowledge and reasoning skills of OR experts when confronted with diverse and complex optimization problems. The dataset, developed by OR experts, features real-world optimization problems that demand multistep reasoning to construct their mathematical models. Our evaluations of various open source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral, reveal their modest performance, highlighting a gap in their ability to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs generalization capabilities, offering valuable insights for future research in this area. The dataset and evaluation code are publicly available. |
| title | Evaluating LLM Reasoning in the Operations Research Domain with ORQA |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2412.17874 |