Online and Offline Evaluation in Search Clarification
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914714412384256 |
|---|---|
| author | Tavakoli, Leila Trippas, Johanne R. Zamani, Hamed Scholer, Falk Sanderson, Mark |
| author_facet | Tavakoli, Leila Trippas, Johanne R. Zamani, Hamed Scholer, Falk Sanderson, Mark |
| contents | The effectiveness of clarification question models in engaging users within search systems is currently constrained, casting doubt on their overall usefulness. To improve the performance of these models, it is crucial to employ assessment approaches that encompass both real-time feedback from users (online evaluation) and the characteristics of clarification questions evaluated through human assessment (offline evaluation). However, the relationship between online and offline evaluations has been debated in information retrieval. This study aims to investigate how this discordance holds in search clarification. We use user engagement as ground truth and employ several offline labels to investigate to what extent the offline ranked lists of clarification resemble the ideal ranked lists based on online user engagement. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_09180 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Online and Offline Evaluation in Search Clarification Tavakoli, Leila Trippas, Johanne R. Zamani, Hamed Scholer, Falk Sanderson, Mark Information Retrieval The effectiveness of clarification question models in engaging users within search systems is currently constrained, casting doubt on their overall usefulness. To improve the performance of these models, it is crucial to employ assessment approaches that encompass both real-time feedback from users (online evaluation) and the characteristics of clarification questions evaluated through human assessment (offline evaluation). However, the relationship between online and offline evaluations has been debated in information retrieval. This study aims to investigate how this discordance holds in search clarification. We use user engagement as ground truth and employ several offline labels to investigate to what extent the offline ranked lists of clarification resemble the ideal ranked lists based on online user engagement. |
| title | Online and Offline Evaluation in Search Clarification |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2403.09180 |