Image de couverture de livre

Rethinking the Evaluation of Dialogue Systems: Effects of User Feedback on Crowdworkers and LLMs

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Siro, Clemencia, Aliannejadi, Mohammad, de Rijke, Maarten
Format:	Preprint
Publié:	2024
Sujets:	Information Retrieval Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems
par: Siro, Clemencia, et autres
Publié: (2024)

AGENT-CQ: Automatic Generation and Evaluation of Clarifying Questions for Conversational Search with LLMs
par: Siro, Clemencia, et autres
Publié: (2024)

Do Images Clarify? A Study on the Effect of Images on Clarifying Questions in Conversational Search
par: Siro, Clemencia, et autres
Publié: (2026)

Multi-Turn Multi-Modal Question Clarification for Enhanced Conversational Understanding
par: Ramezan, Kimia, et autres
Publié: (2025)

Learning to Judge: LLMs Designing and Applying Evaluation Rubrics
par: Siro, Clemencia, et autres
Publié: (2026)

Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement
par: Mousavian, Maryam, et autres
Publié: (2025)

Ranked List Truncation for Large Language Model-based Re-Ranking
par: Meng, Chuan, et autres
Publié: (2024)

Query Performance Prediction using Relevance Judgments Generated by Large Language Models
par: Meng, Chuan, et autres
Publié: (2024)

Multi-Step Semantic Reasoning in Generative Retrieval
par: Dong, Steven, et autres
Publié: (2026)

Understanding Visual Saliency of Outlier Items in Product Search
par: Sarvi, Fatemeh, et autres
Publié: (2025)

Repeat-bias-aware Optimization of Beyond-accuracy Metrics for Next Basket Recommendation
par: Liu, Yuanna, et autres
Publié: (2025)

Asking Multimodal Clarifying Questions in Mixed-Initiative Conversational Search
par: Yuan, Yifei, et autres
Publié: (2024)

IRLab@iKAT24: Learned Sparse Retrieval with Multi-aspect LLM Query Generation for Conversational Search
par: Lupart, Simon, et autres
Publié: (2024)

DiSCo: LLM Knowledge Distillation for Efficient Sparse Retrieval in Conversational Search
par: Lupart, Simon, et autres
Publié: (2024)

ChatR1: Reinforcement Learning for Conversational Reasoning and Retrieval Augmented Question Answering
par: Lupart, Simon, et autres
Publié: (2025)

Information Discovery in e-Commerce
par: Ren, Zhaochun, et autres
Publié: (2024)

A Comprehensive Taxonomy of Negation for NLP and Neural Retrievers
par: Petcu, Roxana, et autres
Publié: (2025)

Re-Rankers as Relevance Judges
par: Meng, Chuan, et autres
Publié: (2026)

Real World Conversational Entity Linking Requires More Than Zeroshots
par: Hoveyda, Mohanna, et autres
Publié: (2024)

SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval
par: Petcu, Roxana, et autres
Publié: (2026)

Can We Use Large Language Models to Fill Relevance Judgment Holes?
par: Abbasiantaeb, Zahra, et autres
Publié: (2024)

A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation
par: Zhang, Weijia, et autres
Publié: (2024)

Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics
par: Zhang, Weijia, et autres
Publié: (2024)

Conversational Search: From Fundamentals to Frontiers in the LLM Era
par: Mo, Fengran, et autres
Publié: (2025)

LLMJudge: LLMs for Relevance Judgments
par: Rahmani, Hossein A., et autres
Publié: (2024)

Hypencoder Revisited: Reproducibility and Analysis of Non-Linear Scoring for First-Stage Retrieval
par: Eichholtz, Arne, et autres
Publié: (2026)

A Cooperative Multi-Agent Framework for Zero-Shot Named Entity Recognition
par: Wang, Zihan, et autres
Publié: (2025)

Cognitive Biases in Large Language Models for News Recommendation
par: Lyu, Yougang, et autres
Publié: (2024)

TREC iKAT 2023: A Test Collection for Evaluating Conversational and Interactive Knowledge Assistants
par: Aliannejadi, Mohammad, et autres
Publié: (2024)

Tutorial on Reasoning for IR & IR for Reasoning
par: Hoveyda, Mohanna, et autres
Publié: (2026)

CorpusBrain++: A Continual Generative Pre-Training Framework for Knowledge-Intensive Language Tasks
par: Guo, Jiafeng, et autres
Publié: (2024)

Investigating LLM Variability in Personalized Conversational Information Retrieval
par: Lupart, Simon, et autres
Publié: (2025)

Report on the 1st Workshop on Large Language Model for Evaluation in Information Retrieval (LLM4Eval 2024) at SIGIR 2024
par: Rahmani, Hossein A., et autres
Publié: (2024)

Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems
par: Zhao, Yilun, et autres
Publié: (2026)

TREC iKAT 2023: The Interactive Knowledge Assistance Track Overview
par: Aliannejadi, Mohammad, et autres
Publié: (2024)

Are We Really Achieving Better Beyond-Accuracy Performance in Next Basket Recommendation?
par: Li, Ming, et autres
Publié: (2024)

A Systematic Study of Pseudo-Relevance Feedback with LLMs
par: Jedidi, Nour, et autres
Publié: (2026)

CAUSE: Counterfactual Assessment of User Satisfaction Estimation in Task-Oriented Dialogue Systems
par: Abolghasemi, Amin, et autres
Publié: (2024)

Query Augmentation by Decoding Semantics from Brain Signals
par: Ye, Ziyi, et autres
Publié: (2024)

Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
par: Wang, Xiaolei, et autres
Publié: (2023)