Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.23009 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912536815730688 |
|---|---|
| author | Chen, Jian Ma, Wenye Liu, Penghang Wang, Wei Song, Tengwei Li, Ming Wang, Chenguang Qin, Jiayu Zhang, Ruiyi Chen, Changyou |
| author_facet | Chen, Jian Ma, Wenye Liu, Penghang Wang, Wei Song, Tengwei Li, Ming Wang, Chenguang Qin, Jiayu Zhang, Ruiyi Chen, Changyou |
| contents | Multimodal Large Language Models (MLLMs) have achieved remarkable visual reasoning abilities in natural images, text-rich documents, and graphic designs. However, their ability to interpret music sheets remains underexplored. To bridge this gap, we introduce MusiXQA, the first comprehensive dataset for evaluating and advancing MLLMs in music sheet understanding. MusiXQA features high-quality synthetic music sheets generated via MusiXTeX, with structured annotations covering note pitch and duration, chords, clefs, key/time signatures, and text, enabling diverse visual QA tasks. Through extensive evaluations, we reveal significant limitations of current state-of-the-art MLLMs in this domain. Beyond benchmarking, we developed Phi-3-MusiX, an MLLM fine-tuned on our dataset, achieving significant performance gains over GPT-based methods. The proposed dataset and model establish a foundation for future advances in MLLMs for music sheet understanding. Code, data, and model will be released upon acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_23009 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models Chen, Jian Ma, Wenye Liu, Penghang Wang, Wei Song, Tengwei Li, Ming Wang, Chenguang Qin, Jiayu Zhang, Ruiyi Chen, Changyou Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) have achieved remarkable visual reasoning abilities in natural images, text-rich documents, and graphic designs. However, their ability to interpret music sheets remains underexplored. To bridge this gap, we introduce MusiXQA, the first comprehensive dataset for evaluating and advancing MLLMs in music sheet understanding. MusiXQA features high-quality synthetic music sheets generated via MusiXTeX, with structured annotations covering note pitch and duration, chords, clefs, key/time signatures, and text, enabling diverse visual QA tasks. Through extensive evaluations, we reveal significant limitations of current state-of-the-art MLLMs in this domain. Beyond benchmarking, we developed Phi-3-MusiX, an MLLM fine-tuned on our dataset, achieving significant performance gains over GPT-based methods. The proposed dataset and model establish a foundation for future advances in MLLMs for music sheet understanding. Code, data, and model will be released upon acceptance. |
| title | MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2506.23009 |