EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917541793759232 |
|---|---|
| author | Bogavelli, Tara Melançon, Gabrielle Gauthier Stankiewicz, Katrina Bamgbose, Oluwanifemi Riols, Fanny Nguyen, Hoang H. Mehndiratta, Raghav Brin, Lindsay Devon Marinier, Joseph Subramani, Hari Madamala, Anil Nemala, Sridhar Krishna Sunkara, Srinivas |
| author_facet | Bogavelli, Tara Melançon, Gabrielle Gauthier Stankiewicz, Katrina Bamgbose, Oluwanifemi Riols, Fanny Nguyen, Hoang H. Mehndiratta, Raghav Brin, Lindsay Devon Marinier, Joseph Subramani, Hari Madamala, Anil Nemala, Sridhar Krishna Sunkara, Srinivas |
| contents | Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to all major agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean $Δ$ up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13841 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents Bogavelli, Tara Melançon, Gabrielle Gauthier Stankiewicz, Katrina Bamgbose, Oluwanifemi Riols, Fanny Nguyen, Hoang H. Mehndiratta, Raghav Brin, Lindsay Devon Marinier, Joseph Subramani, Hari Madamala, Anil Nemala, Sridhar Krishna Sunkara, Srinivas Sound Artificial Intelligence Computation and Language Machine Learning Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to all major agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean $Δ$ up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license. |
| title | EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents |
| topic | Sound Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2605.13841 |