Late Fusion and Multi-Level Fission Amplify Cross-Modal Transfer in Text-Speech LMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908601001443328 |
|---|---|
| author | Cuervo, Santiago Moumen, Adel Labrak, Yanis Khurana, Sameer Laurent, Antoine Rouvier, Mickael Woodland, Phil Marxer, Ricard |
| author_facet | Cuervo, Santiago Moumen, Adel Labrak, Yanis Khurana, Sameer Laurent, Antoine Rouvier, Mickael Woodland, Phil Marxer, Ricard |
| contents | Text-Speech Language Models (TSLMs) -- language models trained to jointly process and generate text and speech -- are commonly trained through an early modality fusion/fission approach, in which both modalities are fed and predicted from a shared backbone via linear layers. We hypothesize that this approach limits cross-modal transfer by neglecting feature compositionality -- specifically, the finer-grained nature of speech representations compared to text -- preventing the emergence of a shared feature hierarchy within model layers. In this paper, we argue that this limitation can be addressed through late fusion and fission, with a fission process that accesses both high- and low-level features for speech generation. Our models implementing these principles, SmolTolk, rival or surpass state-of-the-art TSLMs trained with orders of magnitude more compute, and achieve significantly improved cross-modal performance relative to early fusion/fission baselines. Representation analyses further suggest that our method enhances the model's ability to abstract higher-level, more semantic features from speech, and leads to increasingly shared representation spaces across layers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06211 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Late Fusion and Multi-Level Fission Amplify Cross-Modal Transfer in Text-Speech LMs Cuervo, Santiago Moumen, Adel Labrak, Yanis Khurana, Sameer Laurent, Antoine Rouvier, Mickael Woodland, Phil Marxer, Ricard Computation and Language Artificial Intelligence Audio and Speech Processing Text-Speech Language Models (TSLMs) -- language models trained to jointly process and generate text and speech -- are commonly trained through an early modality fusion/fission approach, in which both modalities are fed and predicted from a shared backbone via linear layers. We hypothesize that this approach limits cross-modal transfer by neglecting feature compositionality -- specifically, the finer-grained nature of speech representations compared to text -- preventing the emergence of a shared feature hierarchy within model layers. In this paper, we argue that this limitation can be addressed through late fusion and fission, with a fission process that accesses both high- and low-level features for speech generation. Our models implementing these principles, SmolTolk, rival or surpass state-of-the-art TSLMs trained with orders of magnitude more compute, and achieve significantly improved cross-modal performance relative to early fusion/fission baselines. Representation analyses further suggest that our method enhances the model's ability to abstract higher-level, more semantic features from speech, and leads to increasingly shared representation spaces across layers. |
| title | Late Fusion and Multi-Level Fission Amplify Cross-Modal Transfer in Text-Speech LMs |
| topic | Computation and Language Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2503.06211 |