Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.14875 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910536714682368 |
|---|---|
| author | Lei, Shuyu Liu, Lingen Yang, Jiaolong Jiao, Yasen Yang, Yuxiang Yang, Yushu Guo, Xiang |
| author_facet | Lei, Shuyu Liu, Lingen Yang, Jiaolong Jiao, Yasen Yang, Yuxiang Yang, Yushu Guo, Xiang |
| contents | Existing auto-regressive language models have demonstrated a remarkable capability to perform a new task with just a few examples in prompt, without requiring any additional training. In order to extend this capability to a multi-modal setting (i.e. speech and language), this paper introduces the Seal model, an abbreviation for speech language model. It incorporates a novel alignment method, in which Kullback-Leibler divergence loss is performed to train a projector that bridges a frozen speech encoder with a frozen language model decoder. The resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks. Additionally, consistency experiments are conducted to validate its robustness on different pre-trained language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_14875 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Seal: Advancing Speech Language Models to be Few-Shot Learners Lei, Shuyu Liu, Lingen Yang, Jiaolong Jiao, Yasen Yang, Yuxiang Yang, Yushu Guo, Xiang Computation and Language Existing auto-regressive language models have demonstrated a remarkable capability to perform a new task with just a few examples in prompt, without requiring any additional training. In order to extend this capability to a multi-modal setting (i.e. speech and language), this paper introduces the Seal model, an abbreviation for speech language model. It incorporates a novel alignment method, in which Kullback-Leibler divergence loss is performed to train a projector that bridges a frozen speech encoder with a frozen language model decoder. The resulting Seal model exhibits robust performance as a few-shot learner on two speech understanding tasks. Additionally, consistency experiments are conducted to validate its robustness on different pre-trained language models. |
| title | Seal: Advancing Speech Language Models to be Few-Shot Learners |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.14875 |