MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929744103079936 |
|---|---|
| author | Sirdeshmukh, Ved Deshpande, Kaustubh Mols, Johannes Jin, Lifeng Cardona, Ed-Yeremai Lee, Dean Kritz, Jeremy Primack, Willow Yue, Summer Xing, Chen |
| author_facet | Sirdeshmukh, Ved Deshpande, Kaustubh Mols, Johannes Jin, Lifeng Cardona, Ed-Yeremai Lee, Dean Kritz, Jeremy Primack, Willow Yue, Summer Xing, Chen |
| contents | We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_17399 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs Sirdeshmukh, Ved Deshpande, Kaustubh Mols, Johannes Jin, Lifeng Cardona, Ed-Yeremai Lee, Dean Kritz, Jeremy Primack, Willow Yue, Summer Xing, Chen Computation and Language Artificial Intelligence We present MultiChallenge, a pioneering benchmark evaluating large language models (LLMs) on conducting multi-turn conversations with human users, a crucial yet underexamined capability for their applications. MultiChallenge identifies four categories of challenges in multi-turn conversations that are not only common and realistic among current human-LLM interactions, but are also challenging to all current frontier LLMs. All 4 challenges require accurate instruction-following, context allocation, and in-context reasoning at the same time. We also develop LLM as judge with instance-level rubrics to facilitate an automatic evaluation method with fair agreement with experienced human raters. Despite achieving near-perfect scores on existing multi-turn evaluation benchmarks, all frontier models have less than 50% accuracy on MultiChallenge, with the top-performing Claude 3.5 Sonnet (June 2024) achieving just a 41.4% average accuracy. |
| title | MultiChallenge: A Realistic Multi-Turn Conversation Evaluation Benchmark Challenging to Frontier LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.17399 |