From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917030991495168 |
|---|---|
| author | Ambilduke, Kshitij Peters, Ben Sannigrahi, Sonal Keshwani, Anil Lam, Tsz Kin Martins, Bruno Martins, André F. T. Boito, Marcely Zanon |
| author_facet | Ambilduke, Kshitij Peters, Ben Sannigrahi, Sonal Keshwani, Anil Lam, Tsz Kin Martins, Bruno Martins, André F. T. Boito, Marcely Zanon |
| contents | We introduce Spire, a speech-augmented language model (LM) capable of both translating and transcribing speech input from English into 10 other languages as well as translating text input in both language directions. Spire integrates the speech modality into an existing multilingual LM via speech discretization and continued pre-training using only 42.5K hours of speech. In particular, we adopt the pretraining framework of multilingual LMs and treat discretized speech input as an additional translation language. This approach not only equips the model with speech capabilities, but also preserves its strong text-based performance. We achieve this using significantly less data than existing speech LMs, demonstrating that discretized speech input integration as an additional language is feasible during LM adaptation. We make our code and models available to the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10620 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM Ambilduke, Kshitij Peters, Ben Sannigrahi, Sonal Keshwani, Anil Lam, Tsz Kin Martins, Bruno Martins, André F. T. Boito, Marcely Zanon Computation and Language We introduce Spire, a speech-augmented language model (LM) capable of both translating and transcribing speech input from English into 10 other languages as well as translating text input in both language directions. Spire integrates the speech modality into an existing multilingual LM via speech discretization and continued pre-training using only 42.5K hours of speech. In particular, we adopt the pretraining framework of multilingual LMs and treat discretized speech input as an additional translation language. This approach not only equips the model with speech capabilities, but also preserves its strong text-based performance. We achieve this using significantly less data than existing speech LMs, demonstrating that discretized speech input integration as an additional language is feasible during LM adaptation. We make our code and models available to the community. |
| title | From TOWER to SPIRE: Adding the Speech Modality to a Translation-Specialist LLM |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.10620 |