An End-to-End Speech Summarization Using Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911940429742080 |
|---|---|
| author | Shang, Hengchao Li, Zongyao Guo, Jiaxin Li, Shaojun Rao, Zhiqiang Luo, Yuanchang Wei, Daimeng Yang, Hao |
| author_facet | Shang, Hengchao Li, Zongyao Guo, Jiaxin Li, Shaojun Rao, Zhiqiang Luo, Yuanchang Wei, Daimeng Yang, Hao |
| contents | Abstractive Speech Summarization (SSum) aims to generate human-like text summaries from spoken content. It encounters difficulties in handling long speech input and capturing the intricate cross-modal mapping between long speech inputs and short text summaries. Research on large language models (LLMs) and multimodal information fusion has provided new insights for addressing these challenges. In this paper, we propose an end-to-end SSum model that utilizes Q-Former as a connector for the audio-text modality and employs LLMs to generate text summaries directly from speech features. We adopt a multi-stage training approach that includes LLM based ASR and Text Summarization (TSum) tasks as auxiliary tasks. ASR tasks are used to align feature spaces and enhance the LLM's ability to handle longer speech. Then, we utilize a curriculum learning strategy to facilitate the model's transition from TSum to SSum. Finally, our model achieves competitive performance on the How-2 dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02005 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An End-to-End Speech Summarization Using Large Language Model Shang, Hengchao Li, Zongyao Guo, Jiaxin Li, Shaojun Rao, Zhiqiang Luo, Yuanchang Wei, Daimeng Yang, Hao Computation and Language Sound Audio and Speech Processing Abstractive Speech Summarization (SSum) aims to generate human-like text summaries from spoken content. It encounters difficulties in handling long speech input and capturing the intricate cross-modal mapping between long speech inputs and short text summaries. Research on large language models (LLMs) and multimodal information fusion has provided new insights for addressing these challenges. In this paper, we propose an end-to-end SSum model that utilizes Q-Former as a connector for the audio-text modality and employs LLMs to generate text summaries directly from speech features. We adopt a multi-stage training approach that includes LLM based ASR and Text Summarization (TSum) tasks as auxiliary tasks. ASR tasks are used to align feature spaces and enhance the LLM's ability to handle longer speech. Then, we utilize a curriculum learning strategy to facilitate the model's transition from TSum to SSum. Finally, our model achieves competitive performance on the How-2 dataset. |
| title | An End-to-End Speech Summarization Using Large Language Model |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2407.02005 |