SimLab: A Platform for Simulation-based Evaluation of Conversational Information Access Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914110634983424 |
|---|---|
| author | Bernard, Nolwenn Suresh, Sharath Chandra Etagi Balog, Krisztian Zhai, ChengXiang |
| author_facet | Bernard, Nolwenn Suresh, Sharath Chandra Etagi Balog, Krisztian Zhai, ChengXiang |
| contents | Progress in conversational information access (CIA) systems has been hindered by the difficulty of evaluating such systems with reproducible experiments. While user simulation offers a promising solution, the lack of infrastructure and tooling to support this evaluation paradigm remains a significant barrier. To address this gap, we introduce SimLab, the first cloud-based platform providing a centralized solution for the community to benchmark both conversational systems and user simulators in a controlled and reproducible setting. We articulate the requirements for such a platform and propose a general infrastructure to meet them. We then present the design and implementation of an initial version of SimLab and showcase its features through an initial simulation-based evaluation task in conversational movie recommendation. Furthermore, we discuss the platform's sustainability and future opportunities for development, inviting the community to drive further progress in the fields of CIA and user simulation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04888 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SimLab: A Platform for Simulation-based Evaluation of Conversational Information Access Systems Bernard, Nolwenn Suresh, Sharath Chandra Etagi Balog, Krisztian Zhai, ChengXiang Information Retrieval Progress in conversational information access (CIA) systems has been hindered by the difficulty of evaluating such systems with reproducible experiments. While user simulation offers a promising solution, the lack of infrastructure and tooling to support this evaluation paradigm remains a significant barrier. To address this gap, we introduce SimLab, the first cloud-based platform providing a centralized solution for the community to benchmark both conversational systems and user simulators in a controlled and reproducible setting. We articulate the requirements for such a platform and propose a general infrastructure to meet them. We then present the design and implementation of an initial version of SimLab and showcase its features through an initial simulation-based evaluation task in conversational movie recommendation. Furthermore, we discuss the platform's sustainability and future opportunities for development, inviting the community to drive further progress in the fields of CIA and user simulation. |
| title | SimLab: A Platform for Simulation-based Evaluation of Conversational Information Access Systems |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2507.04888 |