Resource-Efficient Reference-Free Evaluation of Audio Captions
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917856330907648 |
|---|---|
| author | Mahfuz, Rehana Guo, Yinyi Visser, Erik |
| author_facet | Mahfuz, Rehana Guo, Yinyi Visser, Erik |
| contents | To establish the trustworthiness of systems that automatically generate text captions for audio, images and video, existing reference-free metrics rely on large pretrained models which are impractical to accommodate in resource-constrained settings. To address this, we propose some metrics to elicit the model's confidence in its own generation. To assess how well these metrics replace correctness measures that leverage reference captions, we test their calibration with correctness measures. We discuss why some of these confidence metrics align better with certain correctness measures. Further, we provide insight into why temperature scaling of confidence metrics is effective. Our main contribution is a suite of well-calibrated lightweight confidence metrics for reference-free evaluation of captions in resource-constrained settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_08489 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Resource-Efficient Reference-Free Evaluation of Audio Captions Mahfuz, Rehana Guo, Yinyi Visser, Erik Multimedia Sound Audio and Speech Processing To establish the trustworthiness of systems that automatically generate text captions for audio, images and video, existing reference-free metrics rely on large pretrained models which are impractical to accommodate in resource-constrained settings. To address this, we propose some metrics to elicit the model's confidence in its own generation. To assess how well these metrics replace correctness measures that leverage reference captions, we test their calibration with correctness measures. We discuss why some of these confidence metrics align better with certain correctness measures. Further, we provide insight into why temperature scaling of confidence metrics is effective. Our main contribution is a suite of well-calibrated lightweight confidence metrics for reference-free evaluation of captions in resource-constrained settings. |
| title | Resource-Efficient Reference-Free Evaluation of Audio Captions |
| topic | Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.08489 |