CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917823516770304 |
|---|---|
| author | Cheng, Yiruo Mao, Kelong Zhao, Ziliang Dong, Guanting Qian, Hongjin Wu, Yongkang Sakai, Tetsuya Wen, Ji-Rong Dou, Zhicheng |
| author_facet | Cheng, Yiruo Mao, Kelong Zhao, Ziliang Dong, Guanting Qian, Hongjin Wu, Yongkang Sakai, Tetsuya Wen, Ji-Rong Dou, Zhicheng |
| contents | Retrieval-Augmented Generation (RAG) has become a powerful paradigm for enhancing large language models (LLMs) through external knowledge retrieval. Despite its widespread attention, existing academic research predominantly focuses on single-turn RAG, leaving a significant gap in addressing the complexities of multi-turn conversations found in real-world applications. To bridge this gap, we introduce CORAL, a large-scale benchmark designed to assess RAG systems in realistic multi-turn conversational settings. CORAL includes diverse information-seeking conversations automatically derived from Wikipedia and tackles key challenges such as open-domain coverage, knowledge intensity, free-form responses, and topic shifts. It supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. We propose a unified framework to standardize various conversational RAG methods and conduct a comprehensive evaluation of these methods on CORAL, demonstrating substantial opportunities for improving existing approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23090 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation Cheng, Yiruo Mao, Kelong Zhao, Ziliang Dong, Guanting Qian, Hongjin Wu, Yongkang Sakai, Tetsuya Wen, Ji-Rong Dou, Zhicheng Information Retrieval Computation and Language Retrieval-Augmented Generation (RAG) has become a powerful paradigm for enhancing large language models (LLMs) through external knowledge retrieval. Despite its widespread attention, existing academic research predominantly focuses on single-turn RAG, leaving a significant gap in addressing the complexities of multi-turn conversations found in real-world applications. To bridge this gap, we introduce CORAL, a large-scale benchmark designed to assess RAG systems in realistic multi-turn conversational settings. CORAL includes diverse information-seeking conversations automatically derived from Wikipedia and tackles key challenges such as open-domain coverage, knowledge intensity, free-form responses, and topic shifts. It supports three core tasks of conversational RAG: passage retrieval, response generation, and citation labeling. We propose a unified framework to standardize various conversational RAG methods and conduct a comprehensive evaluation of these methods on CORAL, demonstrating substantial opportunities for improving existing approaches. |
| title | CORAL: Benchmarking Multi-turn Conversational Retrieval-Augmentation Generation |
| topic | Information Retrieval Computation and Language |
| url | https://arxiv.org/abs/2410.23090 |