Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914833918590976 |
|---|---|
| author | Palaskar, Shruti Rudovic, Oggi Dharur, Sameer Pesce, Florian Krishna, Gautam Sivaraman, Aswin Berkowitz, Jack Abdelaziz, Ahmed Hussen Adya, Saurabh Tewfik, Ahmed |
| author_facet | Palaskar, Shruti Rudovic, Oggi Dharur, Sameer Pesce, Florian Krishna, Gautam Sivaraman, Aswin Berkowitz, Jack Abdelaziz, Ahmed Hussen Adya, Saurabh Tewfik, Ahmed |
| contents | Although Large Language Models (LLMs) have shown promise for human-like conversations, they are primarily pre-trained on text data. Incorporating audio or video improves performance, but collecting large-scale multimodal data and pre-training multimodal LLMs is challenging. To this end, we propose a Fusion Low Rank Adaptation (FLoRA) technique that efficiently adapts a pre-trained unimodal LLM to consume new, previously unseen modalities via low rank adaptation. For device-directed speech detection, using FLoRA, the multimodal LLM achieves 22% relative reduction in equal error rate (EER) over the text-only approach and attains performance parity with its full fine-tuning (FFT) counterpart while needing to tune only a fraction of its parameters. Furthermore, with the newly introduced adapter dropout, FLoRA is robust to missing data, improving over FFT by 20% lower EER and 56% lower false accept rate. The proposed approach scales well for model sizes from 16M to 3B parameters. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09617 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection Palaskar, Shruti Rudovic, Oggi Dharur, Sameer Pesce, Florian Krishna, Gautam Sivaraman, Aswin Berkowitz, Jack Abdelaziz, Ahmed Hussen Adya, Saurabh Tewfik, Ahmed Computation and Language Human-Computer Interaction Audio and Speech Processing Although Large Language Models (LLMs) have shown promise for human-like conversations, they are primarily pre-trained on text data. Incorporating audio or video improves performance, but collecting large-scale multimodal data and pre-training multimodal LLMs is challenging. To this end, we propose a Fusion Low Rank Adaptation (FLoRA) technique that efficiently adapts a pre-trained unimodal LLM to consume new, previously unseen modalities via low rank adaptation. For device-directed speech detection, using FLoRA, the multimodal LLM achieves 22% relative reduction in equal error rate (EER) over the text-only approach and attains performance parity with its full fine-tuning (FFT) counterpart while needing to tune only a fraction of its parameters. Furthermore, with the newly introduced adapter dropout, FLoRA is robust to missing data, improving over FFT by 20% lower EER and 56% lower false accept rate. The proposed approach scales well for model sizes from 16M to 3B parameters. |
| title | Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection |
| topic | Computation and Language Human-Computer Interaction Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.09617 |