LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910682137493504 |
|---|---|
| author | Liu, Langming Zhao, Xiangyu Zhang, Chi Gao, Jingtong Wang, Wanyu Fan, Wenqi Wang, Yiqi He, Ming Liu, Zitao Li, Qing |
| author_facet | Liu, Langming Zhao, Xiangyu Zhang, Chi Gao, Jingtong Wang, Wanyu Fan, Wenqi Wang, Yiqi He, Ming Liu, Zitao Li, Qing |
| contents | Transformer models have achieved remarkable success in sequential recommender systems (SRSs). However, computing the attention matrix in traditional dot-product attention mechanisms results in a quadratic complexity with sequence lengths, leading to high computational costs for long-term sequential recommendation. Motivated by the above observation, we propose a novel L2-Normalized Linear Attention for the Transformer-based Sequential Recommender Systems (LinRec), which theoretically improves efficiency while preserving the learning capabilities of the traditional dot-product attention. Specifically, by thoroughly examining the equivalence conditions of efficient attention mechanisms, we show that LinRec possesses linear complexity while preserving the property of attention mechanisms. In addition, we reveal its latent efficiency properties by interpreting the proposed LinRec mechanism through a statistical lens. Extensive experiments are conducted based on two public benchmark datasets, demonstrating that the combination of LinRec and Transformer models achieves comparable or even superior performance than state-of-the-art Transformer-based SRS models while significantly improving time and memory efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01537 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems Liu, Langming Zhao, Xiangyu Zhang, Chi Gao, Jingtong Wang, Wanyu Fan, Wenqi Wang, Yiqi He, Ming Liu, Zitao Li, Qing Information Retrieval Transformer models have achieved remarkable success in sequential recommender systems (SRSs). However, computing the attention matrix in traditional dot-product attention mechanisms results in a quadratic complexity with sequence lengths, leading to high computational costs for long-term sequential recommendation. Motivated by the above observation, we propose a novel L2-Normalized Linear Attention for the Transformer-based Sequential Recommender Systems (LinRec), which theoretically improves efficiency while preserving the learning capabilities of the traditional dot-product attention. Specifically, by thoroughly examining the equivalence conditions of efficient attention mechanisms, we show that LinRec possesses linear complexity while preserving the property of attention mechanisms. In addition, we reveal its latent efficiency properties by interpreting the proposed LinRec mechanism through a statistical lens. Extensive experiments are conducted based on two public benchmark datasets, demonstrating that the combination of LinRec and Transformer models achieves comparable or even superior performance than state-of-the-art Transformer-based SRS models while significantly improving time and memory efficiency. |
| title | LinRec: Linear Attention Mechanism for Long-term Sequential Recommender Systems |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2411.01537 |