MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918513483972608 |
|---|---|
| author | Ruan, Junhao Abudula, Abudukeyumu Li, Bei Yin, Yongjing Liu, Xinyu Jiao, Kechen Chen, Xin Wang, Jingang Cai, Xunliang Xiao, Tong Zhu, Jingbo |
| author_facet | Ruan, Junhao Abudula, Abudukeyumu Li, Bei Yin, Yongjing Liu, Xinyu Jiao, Kechen Chen, Xin Wang, Jingang Cai, Xunliang Xiao, Tong Zhu, Jingbo |
| contents | Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval-Augmented Generation (RAG) systems. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or rigid, unnatural automated heuristics. To address these challenges, we introduce MTR-Suite, a unified framework for auditing, synthesizing, and benchmarking retrieval. It features: (1) MTR-Eval, an LLM-based auditor quantifying alignment gaps in previous benchmarks; (2) MTR-Pipeline, a multi-agent system using greedy traversal clustering to generate high-fidelity dialogues at 1/400th human cost; and (3) MTR-Bench, a rigorous general-domain benchmark. MTR-Bench mimics production-style challenges (hard topic switching, verbosity), offering superior discriminative power. We make our code and data publicly available to facilitate future research at https://github.com/rangehow/mtr-suite. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20729 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks Ruan, Junhao Abudula, Abudukeyumu Li, Bei Yin, Yongjing Liu, Xinyu Jiao, Kechen Chen, Xin Wang, Jingang Cai, Xunliang Xiao, Tong Zhu, Jingbo Computation and Language Accurate evaluation of conversational retrieval is pivotal for advancing Retrieval-Augmented Generation (RAG) systems. However, existing conversational retrieval benchmarks suffer from costly, sparse human annotation or rigid, unnatural automated heuristics. To address these challenges, we introduce MTR-Suite, a unified framework for auditing, synthesizing, and benchmarking retrieval. It features: (1) MTR-Eval, an LLM-based auditor quantifying alignment gaps in previous benchmarks; (2) MTR-Pipeline, a multi-agent system using greedy traversal clustering to generate high-fidelity dialogues at 1/400th human cost; and (3) MTR-Bench, a rigorous general-domain benchmark. MTR-Bench mimics production-style challenges (hard topic switching, verbosity), offering superior discriminative power. We make our code and data publicly available to facilitate future research at https://github.com/rangehow/mtr-suite. |
| title | MTR-Suite: A Framework for Evaluating and Synthesizing Conversational Retrieval Benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.20729 |