Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.20309 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910848413335552 |
|---|---|
| author | Cappello, Franck Madireddy, Sandeep Underwood, Robert Getty, Neil Chia, Nicholas Lee-Ping Ramachandra, Nesar Nguyen, Josh Keceli, Murat Mallick, Tanwi Li, Zilinghan Ngom, Marieme Zhang, Chenhui Yanguas-Gil, Angel Antoniuk, Evan Kailkhura, Bhavya Tian, Minyang Du, Yufeng Ting, Yuan-Sen Wells, Azton Nicolae, Bogdan Maurya, Avinash Rafique, M. Mustafa Huerta, Eliu Li, Bo Foster, Ian Stevens, Rick |
| author_facet | Cappello, Franck Madireddy, Sandeep Underwood, Robert Getty, Neil Chia, Nicholas Lee-Ping Ramachandra, Nesar Nguyen, Josh Keceli, Murat Mallick, Tanwi Li, Zilinghan Ngom, Marieme Zhang, Chenhui Yanguas-Gil, Angel Antoniuk, Evan Kailkhura, Bhavya Tian, Minyang Du, Yufeng Ting, Yuan-Sen Wells, Azton Nicolae, Bogdan Maurya, Avinash Rafique, M. Mustafa Huerta, Eliu Li, Bo Foster, Ian Stevens, Rick |
| contents | Recent advancements have positioned AI, and particularly Large Language Models (LLMs), as transformative tools for scientific research, capable of addressing complex tasks that require reasoning, problem-solving, and decision-making. Their exceptional capabilities suggest their potential as scientific research assistants but also highlight the need for holistic, rigorous, and domain-specific evaluation to assess effectiveness in real-world scientific applications. This paper describes a multifaceted methodology for Evaluating AI models as scientific Research Assistants (EAIRA) developed at Argonne National Laboratory. This methodology incorporates four primary classes of evaluations. 1) Multiple Choice Questions to assess factual recall; 2) Open Response to evaluate advanced reasoning and problem-solving skills; 3) Lab-Style Experiments involving detailed analysis of capabilities as research assistants in controlled environments; and 4) Field-Style Experiments to capture researcher-LLM interactions at scale in a wide range of scientific domains and applications. These complementary methods enable a comprehensive analysis of LLM strengths and weaknesses with respect to their scientific knowledge, reasoning abilities, and adaptability. Recognizing the rapid pace of LLM advancements, we designed the methodology to evolve and adapt so as to ensure its continued relevance and applicability. This paper describes the methodology state at the end of February 2025. Although developed within a subset of scientific domains, the methodology is designed to be generalizable to a wide range of scientific domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_20309 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants Cappello, Franck Madireddy, Sandeep Underwood, Robert Getty, Neil Chia, Nicholas Lee-Ping Ramachandra, Nesar Nguyen, Josh Keceli, Murat Mallick, Tanwi Li, Zilinghan Ngom, Marieme Zhang, Chenhui Yanguas-Gil, Angel Antoniuk, Evan Kailkhura, Bhavya Tian, Minyang Du, Yufeng Ting, Yuan-Sen Wells, Azton Nicolae, Bogdan Maurya, Avinash Rafique, M. Mustafa Huerta, Eliu Li, Bo Foster, Ian Stevens, Rick Artificial Intelligence Recent advancements have positioned AI, and particularly Large Language Models (LLMs), as transformative tools for scientific research, capable of addressing complex tasks that require reasoning, problem-solving, and decision-making. Their exceptional capabilities suggest their potential as scientific research assistants but also highlight the need for holistic, rigorous, and domain-specific evaluation to assess effectiveness in real-world scientific applications. This paper describes a multifaceted methodology for Evaluating AI models as scientific Research Assistants (EAIRA) developed at Argonne National Laboratory. This methodology incorporates four primary classes of evaluations. 1) Multiple Choice Questions to assess factual recall; 2) Open Response to evaluate advanced reasoning and problem-solving skills; 3) Lab-Style Experiments involving detailed analysis of capabilities as research assistants in controlled environments; and 4) Field-Style Experiments to capture researcher-LLM interactions at scale in a wide range of scientific domains and applications. These complementary methods enable a comprehensive analysis of LLM strengths and weaknesses with respect to their scientific knowledge, reasoning abilities, and adaptability. Recognizing the rapid pace of LLM advancements, we designed the methodology to evolve and adapt so as to ensure its continued relevance and applicability. This paper describes the methodology state at the end of February 2025. Although developed within a subset of scientific domains, the methodology is designed to be generalizable to a wide range of scientific domains. |
| title | EAIRA: Establishing a Methodology for Evaluating AI Models as Scientific Research Assistants |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2502.20309 |