Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.17574 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929230107901952 |
|---|---|
| author | Ralambomihanta, Tokiniaina Raharison Mohammadzadeh, Shahrad Islam, Mohammad Sami Nur Jabbour, Wassim Liang, Laurence |
| author_facet | Ralambomihanta, Tokiniaina Raharison Mohammadzadeh, Shahrad Islam, Mohammad Sami Nur Jabbour, Wassim Liang, Laurence |
| contents | The rapid evolution of Large Language Models (LLMs), epitomized by architectures like GPT-4, has reshaped the landscape of natural language processing. This paper introduces a pioneering approach to address the efficiency concerns associated with LLM pre-training, proposing the use of knowledge distillation for cross-architecture transfer. Leveraging insights from the efficient Hyena mechanism, our method replaces attention heads in transformer models by Hyena, offering a cost-effective alternative to traditional pre-training while confronting the challenge of processing long contextual information, inherent in quadratic attention mechanisms. Unlike conventional compression-focused methods, our technique not only enhances inference speed but also surpasses pre-training in terms of both accuracy and efficiency. In the era of evolving LLMs, our work contributes to the pursuit of sustainable AI solutions, striking a balance between computational power and environmental impact. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_17574 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scavenging Hyena: Distilling Transformers into Long Convolution Models Ralambomihanta, Tokiniaina Raharison Mohammadzadeh, Shahrad Islam, Mohammad Sami Nur Jabbour, Wassim Liang, Laurence Computation and Language Machine Learning The rapid evolution of Large Language Models (LLMs), epitomized by architectures like GPT-4, has reshaped the landscape of natural language processing. This paper introduces a pioneering approach to address the efficiency concerns associated with LLM pre-training, proposing the use of knowledge distillation for cross-architecture transfer. Leveraging insights from the efficient Hyena mechanism, our method replaces attention heads in transformer models by Hyena, offering a cost-effective alternative to traditional pre-training while confronting the challenge of processing long contextual information, inherent in quadratic attention mechanisms. Unlike conventional compression-focused methods, our technique not only enhances inference speed but also surpasses pre-training in terms of both accuracy and efficiency. In the era of evolving LLMs, our work contributes to the pursuit of sustainable AI solutions, striking a balance between computational power and environmental impact. |
| title | Scavenging Hyena: Distilling Transformers into Long Convolution Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2401.17574 |