Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929360360964096 |
|---|---|
| author | Qin, Zhen Shen, Xuyang Li, Dong Sun, Weigao Birchfield, Stan Hartley, Richard Zhong, Yiran |
| author_facet | Qin, Zhen Shen, Xuyang Li, Dong Sun, Weigao Birchfield, Stan Hartley, Richard Zhong, Yiran |
| contents | We present the Linear Complexity Sequence Model (LCSM), a comprehensive solution that unites various sequence modeling techniques with linear complexity, including linear attention, state space model, long convolution, and linear RNN, within a single framework. The goal is to enhance comprehension of these models by analyzing the impact of each component from a cohesive and streamlined viewpoint. Specifically, we segment the modeling processes of these models into three distinct stages: Expand, Oscillation, and Shrink (EOS), with each model having its own specific settings. The Expand stage involves projecting the input signal onto a high-dimensional memory state. This is followed by recursive operations performed on the memory state in the Oscillation stage. Finally, the memory state is projected back to a low-dimensional space in the Shrink stage. We perform comprehensive experiments to analyze the impact of different stage settings on language modeling and retrieval tasks. Our results show that data-driven methods are crucial for the effectiveness of the three stages in language modeling, whereas hand-crafted methods yield better performance in retrieval tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_17383 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective Qin, Zhen Shen, Xuyang Li, Dong Sun, Weigao Birchfield, Stan Hartley, Richard Zhong, Yiran Computation and Language We present the Linear Complexity Sequence Model (LCSM), a comprehensive solution that unites various sequence modeling techniques with linear complexity, including linear attention, state space model, long convolution, and linear RNN, within a single framework. The goal is to enhance comprehension of these models by analyzing the impact of each component from a cohesive and streamlined viewpoint. Specifically, we segment the modeling processes of these models into three distinct stages: Expand, Oscillation, and Shrink (EOS), with each model having its own specific settings. The Expand stage involves projecting the input signal onto a high-dimensional memory state. This is followed by recursive operations performed on the memory state in the Oscillation stage. Finally, the memory state is projected back to a low-dimensional space in the Shrink stage. We perform comprehensive experiments to analyze the impact of different stage settings on language modeling and retrieval tasks. Our results show that data-driven methods are crucial for the effectiveness of the three stages in language modeling, whereas hand-crafted methods yield better performance in retrieval tasks. |
| title | Unlocking the Secrets of Linear Complexity Sequence Model from A Unified Perspective |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2405.17383 |