Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914171547811840 |
|---|---|
| author | Nortje, Leanne Oneata, Dan Pirlogeanu, Gabriel Kamper, Herman |
| author_facet | Nortje, Leanne Oneata, Dan Pirlogeanu, Gabriel Kamper, Herman |
| contents | Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_06013 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings Nortje, Leanne Oneata, Dan Pirlogeanu, Gabriel Kamper, Herman Computation and Language Computer Vision and Pattern Recognition Audio and Speech Processing Given an image query, visually prompted keyword localisation (VPKL) aims to find occurrences of the depicted word in a speech collection. This can be useful when transcriptions are not available for a low-resource language (e.g. if it is unwritten). Previous work showed that VPKL can be performed with a visually grounded speech model trained on paired images and unlabelled speech. But all experiments were done on English. Moreover, transcriptions were used to get positive and negative pairs for the contrastive loss. This paper introduces a few-shot learning scheme to mine pairs automatically without transcriptions. On English, this results in only a small drop in performance. We also - for the first time - consider VPKL on a real low-resource language, Yoruba. While scores are reasonable, here we see a bigger drop in performance compared to using ground truth pairs because the mining is less accurate in Yoruba. |
| title | Improved Visually Prompted Keyword Localisation in Real Low-Resource Settings |
| topic | Computation and Language Computer Vision and Pattern Recognition Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.06013 |