Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.23350 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909812870086656 |
|---|---|
| author | Zhao, Jiahao Li, Yunjia Li, Wei Yoshii, Kazuyoshi |
| author_facet | Zhao, Jiahao Li, Yunjia Li, Wei Yoshii, Kazuyoshi |
| contents | As large language models continue to develop, the feasibility and significance of text-based symbolic music tasks have become increasingly prominent. While symbolic music has been widely used in generation tasks, LLM capabilities in understanding and reasoning about symbolic music remain largely underexplored. To address this gap, we propose ABC-Eval, the first open-source benchmark dedicated to the understanding and instruction-following capabilities in text-based ABC notation scores. It comprises 1,086 test samples spanning 10 sub-tasks, covering scenarios from basic musical syntax comprehension to complex sequence-level reasoning. Such a diverse scope poses substantial challenges to models' ability to handle symbolic music tasks. We evaluated seven state-of-the-art LLMs on ABC-Eval, and the results reveal notable limitations in existing models' symbolic music processing capabilities. Furthermore, the consistent performance of individual baselines across different sub-tasks supports the reliability of our benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_23350 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following Zhao, Jiahao Li, Yunjia Li, Wei Yoshii, Kazuyoshi Sound Artificial Intelligence As large language models continue to develop, the feasibility and significance of text-based symbolic music tasks have become increasingly prominent. While symbolic music has been widely used in generation tasks, LLM capabilities in understanding and reasoning about symbolic music remain largely underexplored. To address this gap, we propose ABC-Eval, the first open-source benchmark dedicated to the understanding and instruction-following capabilities in text-based ABC notation scores. It comprises 1,086 test samples spanning 10 sub-tasks, covering scenarios from basic musical syntax comprehension to complex sequence-level reasoning. Such a diverse scope poses substantial challenges to models' ability to handle symbolic music tasks. We evaluated seven state-of-the-art LLMs on ABC-Eval, and the results reveal notable limitations in existing models' symbolic music processing capabilities. Furthermore, the consistent performance of individual baselines across different sub-tasks supports the reliability of our benchmark. |
| title | ABC-Eval: Benchmarking Large Language Models on Symbolic Music Understanding and Instruction Following |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2509.23350 |