Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913860269637632 |
|---|---|
| author | Lin, Guan-Ting Shivakumar, Prashanth Gurunath Gourav, Aditya Gu, Yile Gandhe, Ankur Lee, Hung-yi Bulyko, Ivan |
| author_facet | Lin, Guan-Ting Shivakumar, Prashanth Gurunath Gourav, Aditya Gu, Yile Gandhe, Ankur Lee, Hung-yi Bulyko, Ivan |
| contents | While textless Spoken Language Models (SLMs) have shown potential in end-to-end speech-to-speech modeling, they still lag behind text-based Large Language Models (LLMs) in terms of semantic coherence and relevance. This work introduces the Align-SLM framework, which leverages preference optimization inspired by Reinforcement Learning with AI Feedback (RLAIF) to enhance the semantic understanding of SLMs. Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO). We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation. Experimental results show that our method achieves state-of-the-art performance for SLMs on most benchmarks, highlighting the importance of preference optimization to improve the semantics of SLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01834 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback Lin, Guan-Ting Shivakumar, Prashanth Gurunath Gourav, Aditya Gu, Yile Gandhe, Ankur Lee, Hung-yi Bulyko, Ivan Computation and Language Audio and Speech Processing While textless Spoken Language Models (SLMs) have shown potential in end-to-end speech-to-speech modeling, they still lag behind text-based Large Language Models (LLMs) in terms of semantic coherence and relevance. This work introduces the Align-SLM framework, which leverages preference optimization inspired by Reinforcement Learning with AI Feedback (RLAIF) to enhance the semantic understanding of SLMs. Our approach generates multiple speech continuations from a given prompt and uses semantic metrics to create preference data for Direct Preference Optimization (DPO). We evaluate the framework using ZeroSpeech 2021 benchmarks for lexical and syntactic modeling, the spoken version of the StoryCloze dataset for semantic coherence, and other speech generation metrics, including the GPT4-o score and human evaluation. Experimental results show that our method achieves state-of-the-art performance for SLMs on most benchmarks, highlighting the importance of preference optimization to improve the semantics of SLMs. |
| title | Align-SLM: Textless Spoken Language Models with Reinforcement Learning from AI Feedback |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2411.01834 |