Whisper-UT: A Unified Translation Framework for Speech and Text
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912595301105664 |
|---|---|
| author | Xiao, Cihan Wiesner, Matthew Chakraborty, Debashish Kriz, Reno Cunningham, Keith Murray, Kenton Duh, Kevin Tavarez-Arce, Luis McNamee, Paul Khudanpur, Sanjeev |
| author_facet | Xiao, Cihan Wiesner, Matthew Chakraborty, Debashish Kriz, Reno Cunningham, Keith Murray, Kenton Duh, Kevin Tavarez-Arce, Luis McNamee, Paul Khudanpur, Sanjeev |
| contents | Encoder-decoder models have achieved remarkable success in speech and text tasks, yet efficiently adapting these models to diverse uni/multi-modal scenarios remains an open challenge. In this paper, we propose Whisper-UT, a unified and efficient framework that leverages lightweight adapters to enable seamless adaptation across tasks, including a multi-modal machine translation (MMT) task that explicitly conditions translation on both speech and source language text inputs. By incorporating ASR hypotheses or ground-truth transcripts as prompts, this approach not only enables the system to process both modalities simultaneously but also enhances speech translation (ST) performance through a 2-stage decoding strategy. We demonstrate our methods using the Whisper model, though in principle they are general and could be applied to similar multitask models. We highlight the effectiveness of cross-modal and cross-task fine-tuning, which improves performance without requiring 3-way parallel data. Our results underscore the flexibility, efficiency, and general applicability of the proposed framework for multi-modal translation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16375 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Whisper-UT: A Unified Translation Framework for Speech and Text Xiao, Cihan Wiesner, Matthew Chakraborty, Debashish Kriz, Reno Cunningham, Keith Murray, Kenton Duh, Kevin Tavarez-Arce, Luis McNamee, Paul Khudanpur, Sanjeev Computation and Language Encoder-decoder models have achieved remarkable success in speech and text tasks, yet efficiently adapting these models to diverse uni/multi-modal scenarios remains an open challenge. In this paper, we propose Whisper-UT, a unified and efficient framework that leverages lightweight adapters to enable seamless adaptation across tasks, including a multi-modal machine translation (MMT) task that explicitly conditions translation on both speech and source language text inputs. By incorporating ASR hypotheses or ground-truth transcripts as prompts, this approach not only enables the system to process both modalities simultaneously but also enhances speech translation (ST) performance through a 2-stage decoding strategy. We demonstrate our methods using the Whisper model, though in principle they are general and could be applied to similar multitask models. We highlight the effectiveness of cross-modal and cross-task fine-tuning, which improves performance without requiring 3-way parallel data. Our results underscore the flexibility, efficiency, and general applicability of the proposed framework for multi-modal translation. |
| title | Whisper-UT: A Unified Translation Framework for Speech and Text |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.16375 |