The Oracle Has Spoken: A Multi-Aspect Evaluation of Dialogue in Pythia
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915504602480640 |
|---|---|
| author | Chen, Zixun Babkin, Petr Gupta, Akshat Anumanchipalli, Gopala Liu, Xiaomo |
| author_facet | Chen, Zixun Babkin, Petr Gupta, Akshat Anumanchipalli, Gopala Liu, Xiaomo |
| contents | Dialogue is one of the landmark abilities of large language models (LLMs). Despite its ubiquity, few studies actually distinguish specific ingredients underpinning dialogue behavior emerging during post-training. We employ a comprehensive suite of model-based metrics, each targeting a distinct fine-grained aspect of dialogue, motivated by linguistic theory. We evaluate how the performance of pre-trained Pythia models changes with respect to each of those dimensions, depending on model size and as a result of supervised fine-tuning on conversational datasets. We observe only a mild impact of raw model size on most metrics, whereas fine-tuning quickly saturates the scores for all but the smallest models tested. Somewhat contrary to our expectations, many metrics show very similar trends, especially if they are all rooted in the same evaluator model, which raises the question of their reliability in measuring a specific dimension. To that end, we conduct additional analyses of score distributions, metric correlations, and term frequencies in generated responses to help explain our observations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16487 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Oracle Has Spoken: A Multi-Aspect Evaluation of Dialogue in Pythia Chen, Zixun Babkin, Petr Gupta, Akshat Anumanchipalli, Gopala Liu, Xiaomo Computation and Language Artificial Intelligence Dialogue is one of the landmark abilities of large language models (LLMs). Despite its ubiquity, few studies actually distinguish specific ingredients underpinning dialogue behavior emerging during post-training. We employ a comprehensive suite of model-based metrics, each targeting a distinct fine-grained aspect of dialogue, motivated by linguistic theory. We evaluate how the performance of pre-trained Pythia models changes with respect to each of those dimensions, depending on model size and as a result of supervised fine-tuning on conversational datasets. We observe only a mild impact of raw model size on most metrics, whereas fine-tuning quickly saturates the scores for all but the smallest models tested. Somewhat contrary to our expectations, many metrics show very similar trends, especially if they are all rooted in the same evaluator model, which raises the question of their reliability in measuring a specific dimension. To that end, we conduct additional analyses of score distributions, metric correlations, and term frequencies in generated responses to help explain our observations. |
| title | The Oracle Has Spoken: A Multi-Aspect Evaluation of Dialogue in Pythia |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.16487 |