Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.09037 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915517908910080 |
|---|---|
| author | Zivanovic, Uros Pilkov, Ivan Cancino-Chacón, Carlos Eduardo |
| author_facet | Zivanovic, Uros Pilkov, Ivan Cancino-Chacón, Carlos Eduardo |
| contents | Visual piano transcription (VPT) is the task of obtaining a symbolic representation of a piano performance from visual information alone (e.g., from a top-down video of the piano keyboard). In this work we propose a VPT system based on the vision transformer (ViT), which surpasses previous methods based on convolutional neural networks (CNNs). Our system is trained on the newly introduced R3 dataset, consisting of ca.~31 hours of synchronized video and MIDI recordings of piano performances. We additionally introduce an approach to predict note offsets, which has not been previously explored in this context. We show that our system outperforms the state-of-the-art on the PianoYT dataset for onset prediction and on the R3 dataset for both onsets and offsets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_09037 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Pay Attention to the Keys: Visual Piano Transcription Using Transformers Zivanovic, Uros Pilkov, Ivan Cancino-Chacón, Carlos Eduardo Computer Vision and Pattern Recognition Visual piano transcription (VPT) is the task of obtaining a symbolic representation of a piano performance from visual information alone (e.g., from a top-down video of the piano keyboard). In this work we propose a VPT system based on the vision transformer (ViT), which surpasses previous methods based on convolutional neural networks (CNNs). Our system is trained on the newly introduced R3 dataset, consisting of ca.~31 hours of synchronized video and MIDI recordings of piano performances. We additionally introduce an approach to predict note offsets, which has not been previously explored in this context. We show that our system outperforms the state-of-the-art on the PianoYT dataset for onset prediction and on the R3 dataset for both onsets and offsets. |
| title | Pay Attention to the Keys: Visual Piano Transcription Using Transformers |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2411.09037 |