Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.24164 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911040560693248 |
|---|---|
| author | Huynh, Ngoc Dung Bouadjenek, Mohamed Reda Razzak, Imran Hacid, Hakim Aryal, Sunil |
| author_facet | Huynh, Ngoc Dung Bouadjenek, Mohamed Reda Razzak, Imran Hacid, Hakim Aryal, Sunil |
| contents | Large vision and language models show strong performance in tasks like image captioning, visual question answering, and retrieval. However, challenges remain in integrating speech, text, and vision into a unified model, especially for spoken tasks. Speech generation methods vary (some produce speech directly), others through text (but their impact on quality is unclear). Evaluation often relies on automatic speech recognition, which may introduce bias. We propose SVLA, a unified speech vision language model based on a transformer architecture that handles multimodal inputs and outputs. We train it on 38.2 million speech text image examples, including 64.1 hours of synthetic speech. We also introduce Speech VQA Accuracy, a new metric for evaluating spoken responses. SVLA improves multimodal understanding and generation by better combining speech, vision, and language. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_24164 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SVLA: A Unified Speech-Vision-Language Assistant with Multimodal Reasoning and Speech Generation Huynh, Ngoc Dung Bouadjenek, Mohamed Reda Razzak, Imran Hacid, Hakim Aryal, Sunil Multimedia Large vision and language models show strong performance in tasks like image captioning, visual question answering, and retrieval. However, challenges remain in integrating speech, text, and vision into a unified model, especially for spoken tasks. Speech generation methods vary (some produce speech directly), others through text (but their impact on quality is unclear). Evaluation often relies on automatic speech recognition, which may introduce bias. We propose SVLA, a unified speech vision language model based on a transformer architecture that handles multimodal inputs and outputs. We train it on 38.2 million speech text image examples, including 64.1 hours of synthetic speech. We also introduce Speech VQA Accuracy, a new metric for evaluating spoken responses. SVLA improves multimodal understanding and generation by better combining speech, vision, and language. |
| title | SVLA: A Unified Speech-Vision-Language Assistant with Multimodal Reasoning and Speech Generation |
| topic | Multimedia |
| url | https://arxiv.org/abs/2503.24164 |