Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918031172567040 |
|---|---|
| author | Manakul, Potsawee Sun, Guangzhi Sirichotedumrong, Warit Tharnpipitchai, Kasima Pipatanakul, Kunat |
| author_facet | Manakul, Potsawee Sun, Guangzhi Sirichotedumrong, Warit Tharnpipitchai, Kasima Pipatanakul, Kunat |
| contents | Audio language models process audio inputs using textual prompts for tasks like speech recognition and audio captioning. Although built on multilingual pre-trained components, most are trained primarily on English, limiting their usability for other languages. This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations. To address this, we explore data mixtures that optimize audio language models for both a target language and English while integrating audio comprehension and speech instruction-following into a unified model. Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data. The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_10999 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models Manakul, Potsawee Sun, Guangzhi Sirichotedumrong, Warit Tharnpipitchai, Kasima Pipatanakul, Kunat Computation and Language Artificial Intelligence Sound Audio and Speech Processing Audio language models process audio inputs using textual prompts for tasks like speech recognition and audio captioning. Although built on multilingual pre-trained components, most are trained primarily on English, limiting their usability for other languages. This paper evaluates audio language models on Thai, a low-resource language, and finds that they lack emergent cross-lingual abilities despite their multilingual foundations. To address this, we explore data mixtures that optimize audio language models for both a target language and English while integrating audio comprehension and speech instruction-following into a unified model. Our experiments provide insights into improving instruction-following in low-resource languages by balancing language-specific and multilingual training data. The proposed model, Typhoon-Audio, significantly outperforms existing open-source models and achieves performance comparable to state-of-the-art Gemini-1.5-Pro in both English and Thai. |
| title | Enhancing Low-Resource Language and Instruction Following Capabilities of Audio Language Models |
| topic | Computation and Language Artificial Intelligence Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.10999 |