Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.21815 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916715322933248 |
|---|---|
| author | Zhang, Huan Liang, Jinhua Phan, Huy Wang, Wenwu Benetos, Emmanouil |
| author_facet | Zhang, Huan Liang, Jinhua Phan, Huy Wang, Wenwu Benetos, Emmanouil |
| contents | Evaluating generative models remains a fundamental challenge, particularly when the goal is to reflect human preferences. In this paper, we use music generation as a case study to investigate the gap between automatic evaluation metrics and human preferences. We conduct comparative experiments across five state-of-the-art music generation approaches, assessing both perceptual quality and distributional similarity to human-composed music. Specifically, we evaluate synthesis music from various perceptual dimensions and examine reference-based metrics such as Mauve Audio Divergence (MAD) and Kernel Audio Distance (KAD). Our findings reveal significant inconsistencies across the different metrics, highlighting the limitation of the current evaluation practice. To support further research, we release a benchmark dataset comprising samples from multiple models. This study provides a broader perspective on the alignment of human preference in generative modeling, advocating for more human-centered evaluation strategies across domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_21815 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Aesthetics to Human Preferences: Comparative Perspectives of Evaluating Text-to-Music Systems Zhang, Huan Liang, Jinhua Phan, Huy Wang, Wenwu Benetos, Emmanouil Audio and Speech Processing Evaluating generative models remains a fundamental challenge, particularly when the goal is to reflect human preferences. In this paper, we use music generation as a case study to investigate the gap between automatic evaluation metrics and human preferences. We conduct comparative experiments across five state-of-the-art music generation approaches, assessing both perceptual quality and distributional similarity to human-composed music. Specifically, we evaluate synthesis music from various perceptual dimensions and examine reference-based metrics such as Mauve Audio Divergence (MAD) and Kernel Audio Distance (KAD). Our findings reveal significant inconsistencies across the different metrics, highlighting the limitation of the current evaluation practice. To support further research, we release a benchmark dataset comprising samples from multiple models. This study provides a broader perspective on the alignment of human preference in generative modeling, advocating for more human-centered evaluation strategies across domains. |
| title | From Aesthetics to Human Preferences: Comparative Perspectives of Evaluating Text-to-Music Systems |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2504.21815 |