Cover Image

Factors in Crowdsourcing for Evaluation of Complex Dialogue Systems

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Aicher, Annalena, Hillmann, Stefan, Feustel, Isabel, Michael, Thilo, Möller, Sebastian, Minker, Wolfgang
Format:	Preprint
Published:	2024
Subjects:	Human-Computer Interaction
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Exploring the Impact of Non-Verbal Virtual Agent Behavior on User Engagement in Argumentative Dialogues
by: Aicher, Annalena Bea, et al.
Published: (2024)

Multi-Intent Recognition in Dialogue Understanding: A Comparison Between Smaller Open-Source LLMs
by: Ahmad, Adnan, et al.
Published: (2025)

CAIM: Development and Evaluation of a Cognitive AI Memory Framework for Long-Term Interaction with Intelligent Agents
by: Westhäußer, Rebecca, et al.
Published: (2025)

Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
by: Westhäußer, Rebecca, et al.
Published: (2025)

Context Does Matter: Implications for Crowdsourced Evaluation Labels in Task-Oriented Dialogue Systems
by: Siro, Clemencia, et al.
Published: (2024)

ARCADE: An Augmented Reality Display Environment for Multimodal Interaction with Conversational Agents
by: Schindler, Carolin, et al.
Published: (2024)

Efficient Online Crowdsourcing with Complex Annotations
by: Meir, Reshef, et al.
Published: (2024)

Evaluating Saliency Explanations in NLP by Crowdsourcing
by: Lu, Xiaotian, et al.
Published: (2024)

Crowdsourcing: A Framework for Usability Evaluation
by: Nasir, Muhammad
Published: (2024)

Towards Fair Pay and Equal Work: Imposing View Time Limits in Crowdsourced Image Classification
by: Lim, Gordon, et al.
Published: (2024)

Efficiently Crowdsourcing Visual Importance with Punch-Hole Annotation
by: Chang, Minsuk, et al.
Published: (2024)

Generating A Crowdsourced Conversation Dataset to Combat Cybergrooming
by: Zhang, Xinyi, et al.
Published: (2024)

IP-Dialog: Evaluating Implicit Personalization in Dialogue Systems with Synthetic Data
by: Peng, Bo, et al.
Published: (2025)

User-Centric Evaluation Methods for Digital Twin Applications in Extended Reality
by: Vona, Francesco, et al.
Published: (2025)

Digital Twins for Extended Reality Tourism: User Experience Evaluation Across User Groups
by: Warsinke, Maximilian, et al.
Published: (2025)

Crowdsourcing Task Traces for Service Robotics
by: Porfirio, David, et al.
Published: (2024)

Grid Labeling: Crowdsourcing Task-Specific Importance from Visualizations
by: Chang, Minsuk, et al.
Published: (2025)

Design and Evaluation of Camera-Centric Mobile Crowdsourcing Applications
by: Stylianou, Abby, et al.
Published: (2024)

Detecting the Use of Generative AI in Crowdsourced Surveys: Implications for Data Integrity
by: Zhang, Dapeng, et al.
Published: (2025)

Free Lunch for User Experience: Crowdsourcing Agents for Scalable User Studies
by: Liu, Siyang, et al.
Published: (2025)

A Collaborative Crowdsourcing Method for Designing External Interfaces for Autonomous Vehicles
by: Cumbal, Ronald, et al.
Published: (2026)

"Being Simple on Complex Issues" -- Accounts on Visual Data Communication about Climate Change
by: Schuster, Regina, et al.
Published: (2022)

Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisations
by: Braggaar, Anouck, et al.
Published: (2023)

Partnering with Generative AI: Experimental Evaluation of Human-Led and Model-Led Interaction in Human-AI Co-Creation
by: Maier, Sebastian, et al.
Published: (2025)

Chatbots to strengthen democracy: An interdisciplinary seminar to train identifying argumentation techniques of science denial
by: Siegert, Ingo, et al.
Published: (2025)

A Methodology for Identifying Evaluation Items for Practical Dialogue Systems Based on Business-Dialogue System Alignment Models
by: Nakano, Mikio, et al.
Published: (2026)

A Multi-Agent Dual Dialogue System to Support Mental Health Care Providers
by: Kampman, Onno P., et al.
Published: (2024)

Better Together? The Role of Explanations in Supporting Novices in Individual and Collective Deliberations about AI
by: Schmude, Timothée, et al.
Published: (2024)

CrowdGenUI: Aligning LLM-Based UI Generation with Crowdsourced User Preferences
by: Liu, Yimeng, et al.
Published: (2024)

Is Crowdsourcing a Puppet Show? Detecting a New Type of Fraud in Online Platforms
by: Wang, Shengqian, et al.
Published: (2025)

Crowdsourcing eHMI Designs: A Participatory Approach to Autonomous Vehicle-Pedestrian Communication
by: Cumbal, Ronald, et al.
Published: (2025)

Improving Data Quality via Pre-Task Participant Screening in Crowdsourced GUI Experiments
by: Miyama, Takaya, et al.
Published: (2026)

Evaluation of a Sign Language Avatar on Comprehensibility, User Experience \& Acceptability
by: Wasserroth, Fenya, et al.
Published: (2025)

Evaluating LLM Adaptation to Sociodemographic Factors: User Profile vs. Dialogue History
by: Zhong, Qishuai, et al.
Published: (2025)

Diversified Ensembling: An Experiment in Crowdsourced Machine Learning
by: Globus-Harris, Ira, et al.
Published: (2024)

An Analysis of User Behaviors for Objectively Evaluating Spoken Dialogue Systems
by: Inoue, Koji, et al.
Published: (2024)

Collaborative Evaluation of Deepfake Text with Deliberation-Enhancing Dialogue Systems
by: Lee, Jooyoung, et al.
Published: (2025)

From Tool to Teammate: LLM Coding Agents as Collaborative Partners for Behavioral Labeling in Educational Dialogue Analysis
by: Chen, Eason, et al.
Published: (2026)

A Perspective on Crowdsourcing and Human-in-the-Loop Workflows in Precision Health
by: Washington, Peter
Published: (2023)

Complex Cognition: A New Theoretical Foundation for the Design and Evaluation of Visual Analytics Systems
by: Zhang, Xiaolong
Published: (2026)