SOLAR: SVD-Optimized Lifelong Attention for Recommendation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908861608230912 |
|---|---|
| author | Zhang, Chenghao Feng, Chao Pu, Yuanhao Yang, Xunyong Yu, Wenhui Li, Xiang Liu, Yongqi Hu, Lantao Zhan, Kaiqiao Li, Han Gai, Kun |
| author_facet | Zhang, Chenghao Feng, Chao Pu, Yuanhao Yang, Xunyong Yu, Wenhui Li, Xiang Liu, Yongqi Hu, Lantao Zhan, Kaiqiao Li, Han Gai, Kun |
| contents | Attention mechanism remains the defining operator in Transformers since it provides expressive global credit assignment, yet its $O(N^2 d)$ time and memory cost in sequence length $N$ makes long-context modeling expensive and often forces truncation or other heuristics. Linear attention reduces complexity to $O(N d^2)$ by reordering computation through kernel feature maps, but this reformulation drops the softmax mechanism and shifts the attention score distribution. In recommender systems, low-rank structure in matrices is not a rare case, but rather the default inductive bias in its representation learning, particularly explicit in the user behavior sequence modeling. Leveraging this structure, we introduce SVD-Attention, which is theoretically lossless on low-rank matrices and preserves softmax while reducing attention complexity from $O(N^2 d)$ to $O(Ndr)$. With SVD-Attention, we propose SOLAR, SVD-Optimized Lifelong Attention for Recommendation, a sequence modeling framework that supports behavior sequences of ten-thousand scale and candidate sets of several thousand items in cascading process without any filtering. In Kuaishou's online recommendation scenario, SOLAR delivers a 0.68\% Video Views gain together with additional business metrics improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_02561 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SOLAR: SVD-Optimized Lifelong Attention for Recommendation Zhang, Chenghao Feng, Chao Pu, Yuanhao Yang, Xunyong Yu, Wenhui Li, Xiang Liu, Yongqi Hu, Lantao Zhan, Kaiqiao Li, Han Gai, Kun Information Retrieval Computer Vision and Pattern Recognition Machine Learning Attention mechanism remains the defining operator in Transformers since it provides expressive global credit assignment, yet its $O(N^2 d)$ time and memory cost in sequence length $N$ makes long-context modeling expensive and often forces truncation or other heuristics. Linear attention reduces complexity to $O(N d^2)$ by reordering computation through kernel feature maps, but this reformulation drops the softmax mechanism and shifts the attention score distribution. In recommender systems, low-rank structure in matrices is not a rare case, but rather the default inductive bias in its representation learning, particularly explicit in the user behavior sequence modeling. Leveraging this structure, we introduce SVD-Attention, which is theoretically lossless on low-rank matrices and preserves softmax while reducing attention complexity from $O(N^2 d)$ to $O(Ndr)$. With SVD-Attention, we propose SOLAR, SVD-Optimized Lifelong Attention for Recommendation, a sequence modeling framework that supports behavior sequences of ten-thousand scale and candidate sets of several thousand items in cascading process without any filtering. In Kuaishou's online recommendation scenario, SOLAR delivers a 0.68\% Video Views gain together with additional business metrics improvements. |
| title | SOLAR: SVD-Optimized Lifelong Attention for Recommendation |
| topic | Information Retrieval Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2603.02561 |