BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912401326080000 |
|---|---|
| author | Nishimoto, Tomohiro Nishimura, Taichi Yamamoto, Koki Shirai, Keisuke Kameko, Hirotaka Haneji, Yuto Yoshida, Tomoya Kajimura, Keiya Cui, Taiyu Nishiwaki, Chihiro Daikoku, Eriko Okuda, Natsuko Ono, Fumihito Mori, Shinsuke |
| author_facet | Nishimoto, Tomohiro Nishimura, Taichi Yamamoto, Koki Shirai, Keisuke Kameko, Hirotaka Haneji, Yuto Yoshida, Tomoya Kajimura, Keiya Cui, Taiyu Nishiwaki, Chihiro Daikoku, Eriko Okuda, Natsuko Ono, Fumihito Mori, Shinsuke |
| contents | This paper introduces BioVL-QR, a biochemical vision-and-language dataset comprising 23 egocentric experiment videos, corresponding protocols, and vision-and-language alignments. A major challenge in understanding biochemical videos is detecting equipment, reagents, and containers because of the cluttered environment and indistinguishable objects. Previous studies assumed manual object annotation, which is costly and time-consuming. To address the issue, we focus on Micro QR Codes. However, detecting objects using only Micro QR Codes is still difficult due to blur and occlusion caused by object manipulation. To overcome this, we propose an object labeling method combining a Micro QR Code detector with an off-the-shelf hand object detector. As an application of the method and BioVL-QR, we tackled the task of localizing the procedural steps in an instructional video. The experimental results show that using Micro QR Codes and our method improves biochemical video understanding. Data and code are available through https://nishi10mo.github.io/BioVL-QR/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03161 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes Nishimoto, Tomohiro Nishimura, Taichi Yamamoto, Koki Shirai, Keisuke Kameko, Hirotaka Haneji, Yuto Yoshida, Tomoya Kajimura, Keiya Cui, Taiyu Nishiwaki, Chihiro Daikoku, Eriko Okuda, Natsuko Ono, Fumihito Mori, Shinsuke Computer Vision and Pattern Recognition Computation and Language Multimedia This paper introduces BioVL-QR, a biochemical vision-and-language dataset comprising 23 egocentric experiment videos, corresponding protocols, and vision-and-language alignments. A major challenge in understanding biochemical videos is detecting equipment, reagents, and containers because of the cluttered environment and indistinguishable objects. Previous studies assumed manual object annotation, which is costly and time-consuming. To address the issue, we focus on Micro QR Codes. However, detecting objects using only Micro QR Codes is still difficult due to blur and occlusion caused by object manipulation. To overcome this, we propose an object labeling method combining a Micro QR Code detector with an off-the-shelf hand object detector. As an application of the method and BioVL-QR, we tackled the task of localizing the procedural steps in an instructional video. The experimental results show that using Micro QR Codes and our method improves biochemical video understanding. Data and code are available through https://nishi10mo.github.io/BioVL-QR/ |
| title | BioVL-QR: Egocentric Biochemical Vision-and-Language Dataset Using Micro QR Codes |
| topic | Computer Vision and Pattern Recognition Computation and Language Multimedia |
| url | https://arxiv.org/abs/2404.03161 |