CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908361329475584 |
|---|---|
| author | Cui, Hao Shamsi, Zahra Cheon, Gowoon Ma, Xuejian Li, Shutong Tikhanovskaya, Maria Norgaard, Peter Mudur, Nayantara Plomecka, Martyna Raccuglia, Paul Bahri, Yasaman Albert, Victor V. Srinivasan, Pranesh Pan, Haining Faist, Philippe Rohr, Brian Cubuk, Ekin Dogus Aykol, Muratahan Merchant, Amil Statt, Michael J. Morris, Dan Purves, Drew Kleeman, Elise Alcantara, Ruth Abraham, Matthew Mohammad, Muqthar VanLee, Ean Phing Jiang, Chenfei Dorfman, Elizabeth Kim, Eun-Ah Brenner, Michael P Jain, Viren Ponda, Sameera Venugopalan, Subhashini |
| author_facet | Cui, Hao Shamsi, Zahra Cheon, Gowoon Ma, Xuejian Li, Shutong Tikhanovskaya, Maria Norgaard, Peter Mudur, Nayantara Plomecka, Martyna Raccuglia, Paul Bahri, Yasaman Albert, Victor V. Srinivasan, Pranesh Pan, Haining Faist, Philippe Rohr, Brian Cubuk, Ekin Dogus Aykol, Muratahan Merchant, Amil Statt, Michael J. Morris, Dan Purves, Drew Kleeman, Elise Alcantara, Ruth Abraham, Matthew Mohammad, Muqthar VanLee, Ean Phing Jiang, Chenfei Dorfman, Elizabeth Kim, Eun-Ah Brenner, Michael P Jain, Viren Ponda, Sameera Venugopalan, Subhashini |
| contents | Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning. While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Evaluation code and data are in https://github.com/google/curie |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_13517 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning Cui, Hao Shamsi, Zahra Cheon, Gowoon Ma, Xuejian Li, Shutong Tikhanovskaya, Maria Norgaard, Peter Mudur, Nayantara Plomecka, Martyna Raccuglia, Paul Bahri, Yasaman Albert, Victor V. Srinivasan, Pranesh Pan, Haining Faist, Philippe Rohr, Brian Cubuk, Ekin Dogus Aykol, Muratahan Merchant, Amil Statt, Michael J. Morris, Dan Purves, Drew Kleeman, Elise Alcantara, Ruth Abraham, Matthew Mohammad, Muqthar VanLee, Ean Phing Jiang, Chenfei Dorfman, Elizabeth Kim, Eun-Ah Brenner, Michael P Jain, Viren Ponda, Sameera Venugopalan, Subhashini Computation and Language Artificial Intelligence Scientific problem-solving involves synthesizing information while applying expert knowledge. We introduce CURIE, a scientific long-Context Understanding,Reasoning and Information Extraction benchmark to measure the potential of Large Language Models (LLMs) in scientific problem-solving and assisting scientists in realistic workflows. This benchmark introduces ten challenging tasks with a total of 580 problems and solution pairs curated by experts in six disciplines - materials science, condensed matter physics, quantum computing, geospatial analysis, biodiversity, and proteins - covering both experimental and theoretical work-flows in science. We evaluate a range of closed and open LLMs on tasks in CURIE which requires domain expertise, comprehension of long in-context information,and multi-step reasoning. While Gemini Flash 2.0 and Claude-3 show consistent high comprehension across domains, the popular GPT-4o and command-R+ fail dramatically on protein sequencing tasks. With the best performance at 32% there is much room for improvement for all models. We hope that insights gained from CURIE can guide the future development of LLMs in sciences. Evaluation code and data are in https://github.com/google/curie |
| title | CURIE: Evaluating LLMs On Multitask Scientific Long Context Understanding and Reasoning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.13517 |