Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.11597 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909903165063168 |
|---|---|
| author | Huebscher, Michelle Chen Mach, Katharine Stanić, Aleksandar Leippold, Markus Gaiarin, Ben Hausfather, Zeke Rawat, Elisa Fischer, Erich Ciaramita, Massimiliano Rogelj, Joeri Buck, Christian Saralegui, Lierni Sestorain Knutti, Reto |
| author_facet | Huebscher, Michelle Chen Mach, Katharine Stanić, Aleksandar Leippold, Markus Gaiarin, Ben Hausfather, Zeke Rawat, Elisa Fischer, Erich Ciaramita, Massimiliano Rogelj, Joeri Buck, Christian Saralegui, Lierni Sestorain Knutti, Reto |
| contents | Evaluating how Large Language Models (LLMs) handle complex, specialized knowledge remains a critical challenge. We address this through the lens of climate change by introducing CLINB, a benchmark that assesses models on open-ended, grounded, multimodal question answering tasks with clear requirements for knowledge quality and evidential support. CLINB relies on a dataset of real users' questions and evaluation rubrics curated by leading climate scientists. We implement and validate a model-based evaluation process and evaluate several frontier models. Our findings reveal a critical dichotomy. Frontier models demonstrate remarkable knowledge synthesis capabilities, often exhibiting PhD-level understanding and presentation quality. They outperform "hybrid" answers curated by domain experts assisted by weaker models. However, this performance is countered by failures in grounding. The quality of evidence varies, with substantial hallucination rates for references and images. We argue that bridging this gap between knowledge synthesis and verifiable attribution is essential for the deployment of AI in scientific workflows and that reliable, interpretable benchmarks like CLINB are needed to progress towards building trustworthy AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_11597 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CLINB: A Climate Intelligence Benchmark for Foundational Models Huebscher, Michelle Chen Mach, Katharine Stanić, Aleksandar Leippold, Markus Gaiarin, Ben Hausfather, Zeke Rawat, Elisa Fischer, Erich Ciaramita, Massimiliano Rogelj, Joeri Buck, Christian Saralegui, Lierni Sestorain Knutti, Reto Artificial Intelligence Computation and Language Evaluating how Large Language Models (LLMs) handle complex, specialized knowledge remains a critical challenge. We address this through the lens of climate change by introducing CLINB, a benchmark that assesses models on open-ended, grounded, multimodal question answering tasks with clear requirements for knowledge quality and evidential support. CLINB relies on a dataset of real users' questions and evaluation rubrics curated by leading climate scientists. We implement and validate a model-based evaluation process and evaluate several frontier models. Our findings reveal a critical dichotomy. Frontier models demonstrate remarkable knowledge synthesis capabilities, often exhibiting PhD-level understanding and presentation quality. They outperform "hybrid" answers curated by domain experts assisted by weaker models. However, this performance is countered by failures in grounding. The quality of evidence varies, with substantial hallucination rates for references and images. We argue that bridging this gap between knowledge synthesis and verifiable attribution is essential for the deployment of AI in scientific workflows and that reliable, interpretable benchmarks like CLINB are needed to progress towards building trustworthy AI systems. |
| title | CLINB: A Climate Intelligence Benchmark for Foundational Models |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2511.11597 |