Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914590766399488 |
|---|---|
| author | Li, Guoming Zhang, Shangyu Pan, Junwei Ning, Wentao Chen, Jin Xue, Gengsheng Zhou, Chao Huang, Shudong Gu, Haijie Yang, Menglin |
| author_facet | Li, Guoming Zhang, Shangyu Pan, Junwei Ning, Wentao Chen, Jin Xue, Gengsheng Zhou, Chao Huang, Shudong Gu, Haijie Yang, Menglin |
| contents | Scaling recommendation models is a central challenge in recommender systems. Recently, RankMixer has emerged as an effective solution, operating on a unified token representation and alternating between token mixing and per-token feedforward networks (P-FFNs) to achieve scalable performance. However, RankMixer suffers from \textit{embedding collapse}, where learned representations have low effective rank, limiting expressivity and underutilizing the expanded representation space. Through empirical analysis and theoretical insights, we identify rigid token mixing and P-FFN modules as the primary causes of this phenomenon, jointly inducing a \textbf{damped oscillatory trajectory} in effective-rank evolution across layers. To address it, we propose RankElastor, a novel architecture that produces spectrum-robust representations with provable collapse mitigation. RankElastor introduces two components: (i) \textbf{parameterized full mixing}, which enables expressive token mixing with improved spectral robustness; and (ii) \textbf{GLU-improved P-FFNs}, which stabilize representation spectra through GLU-style FFN modules. Extensive experiments on large-scale industrial datasets demonstrate that RankElastor consistently improves recommendation performance, mitigates embedding collapse, and exhibits robust scaling behavior. Code is available at this GitHub repository: https://github.com/vasile-paskardlgm/RankElastor |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_23191 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation Li, Guoming Zhang, Shangyu Pan, Junwei Ning, Wentao Chen, Jin Xue, Gengsheng Zhou, Chao Huang, Shudong Gu, Haijie Yang, Menglin Machine Learning Information Retrieval Numerical Analysis Scaling recommendation models is a central challenge in recommender systems. Recently, RankMixer has emerged as an effective solution, operating on a unified token representation and alternating between token mixing and per-token feedforward networks (P-FFNs) to achieve scalable performance. However, RankMixer suffers from \textit{embedding collapse}, where learned representations have low effective rank, limiting expressivity and underutilizing the expanded representation space. Through empirical analysis and theoretical insights, we identify rigid token mixing and P-FFN modules as the primary causes of this phenomenon, jointly inducing a \textbf{damped oscillatory trajectory} in effective-rank evolution across layers. To address it, we propose RankElastor, a novel architecture that produces spectrum-robust representations with provable collapse mitigation. RankElastor introduces two components: (i) \textbf{parameterized full mixing}, which enables expressive token mixing with improved spectral robustness; and (ii) \textbf{GLU-improved P-FFNs}, which stabilize representation spectra through GLU-style FFN modules. Extensive experiments on large-scale industrial datasets demonstrate that RankElastor consistently improves recommendation performance, mitigates embedding collapse, and exhibits robust scaling behavior. Code is available at this GitHub repository: https://github.com/vasile-paskardlgm/RankElastor |
| title | Expand More, Shrink Less: Shaping Effective-Rank Dynamics for Dense Scaling in Recommendation |
| topic | Machine Learning Information Retrieval Numerical Analysis |
| url | https://arxiv.org/abs/2605.23191 |