Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916263204225024 |
|---|---|
| author | Kong, Zhifeng Goel, Arushi Badlani, Rohan Ping, Wei Valle, Rafael Catanzaro, Bryan |
| author_facet | Kong, Zhifeng Goel, Arushi Badlani, Rohan Ping, Wei Valle, Rafael Catanzaro, Bryan |
| contents | Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series of training techniques, architecture design, and data strategies to enhance our model with these abilities. Extensive evaluations across various audio understanding tasks confirm the efficacy of our method, setting new state-of-the-art benchmarks. Our demo website is https://audioflamingo.github.io/ and the code is open-sourced at https://github.com/NVIDIA/audio-flamingo. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01831 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities Kong, Zhifeng Goel, Arushi Badlani, Rohan Ping, Wei Valle, Rafael Catanzaro, Bryan Sound Machine Learning Audio and Speech Processing Augmenting large language models (LLMs) to understand audio -- including non-speech sounds and non-verbal speech -- is critically important for diverse real-world applications of LLMs. In this paper, we propose Audio Flamingo, a novel audio language model with 1) strong audio understanding abilities, 2) the ability to quickly adapt to unseen tasks via in-context learning and retrieval, and 3) strong multi-turn dialogue abilities. We introduce a series of training techniques, architecture design, and data strategies to enhance our model with these abilities. Extensive evaluations across various audio understanding tasks confirm the efficacy of our method, setting new state-of-the-art benchmarks. Our demo website is https://audioflamingo.github.io/ and the code is open-sourced at https://github.com/NVIDIA/audio-flamingo. |
| title | Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities |
| topic | Sound Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2402.01831 |