UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914615247503360 |
|---|---|
| author | Jia, Qi Zhao, Haodong Pei, Dun Song, Xiujie Shen, Ye Wang, Shibo Chen, Zijian Zhang, Zicheng Zhu, Xiangyang Zhai, Guangtao |
| author_facet | Jia, Qi Zhao, Haodong Pei, Dun Song, Xiujie Shen, Ye Wang, Shibo Chen, Zijian Zhang, Zicheng Zhu, Xiangyang Zhai, Guangtao |
| contents | Benchmarking large language models (LLMs) and agents in multi-turn interactive scenarios is essential for understanding their practical capabilities. However, existing evaluation protocols are highly heterogeneous, differing significantly in dataset formats, model interfaces, and evaluation pipelines, which severely impedes systematic comparison. In this work, we present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems. The core contribution of UDE lies in its holistic unification: it standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a hierarchical scoring aggregation. It also supports efficient large-scale evaluation through parallel generation and scoring, as well as checkpoint resume to eliminate redundant computation. Leveraging UDE, we conduct an extensive evaluation across diverse multi-dimensional benchmarks. Our empirical analysis shows that no single system consistently outperforms others across all benchmarks, while current memory agents often fail to surpass full-context baselines. Further analyses highlight several future directions, including benchmark deduplication and more adaptive memory architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_23160 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities Jia, Qi Zhao, Haodong Pei, Dun Song, Xiujie Shen, Ye Wang, Shibo Chen, Zijian Zhang, Zicheng Zhu, Xiangyang Zhai, Guangtao Computation and Language Benchmarking large language models (LLMs) and agents in multi-turn interactive scenarios is essential for understanding their practical capabilities. However, existing evaluation protocols are highly heterogeneous, differing significantly in dataset formats, model interfaces, and evaluation pipelines, which severely impedes systematic comparison. In this work, we present UniDial-EvalKit (UDE), a unified evaluation toolkit for assessing interactive AI systems. The core contribution of UDE lies in its holistic unification: it standardizes heterogeneous data formats into a universal schema, streamlines complex evaluation pipelines through a modular architecture, and aligns metric calculations under a hierarchical scoring aggregation. It also supports efficient large-scale evaluation through parallel generation and scoring, as well as checkpoint resume to eliminate redundant computation. Leveraging UDE, we conduct an extensive evaluation across diverse multi-dimensional benchmarks. Our empirical analysis shows that no single system consistently outperforms others across all benchmarks, while current memory agents often fail to surpass full-context baselines. Further analyses highlight several future directions, including benchmark deduplication and more adaptive memory architectures. |
| title | UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.23160 |