RankMixer: Scaling Up Ranking Models in Industrial Recommenders
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916864567803904 |
|---|---|
| author | Zhu, Jie Fan, Zhifang Zhu, Xiaoxie Jiang, Yuchen Wang, Hangyu Han, Xintian Ding, Haoran Wang, Xinmin Zhao, Wenlin Gong, Zhen Yang, Huizhi Chai, Zheng Chen, Zhe Zheng, Yuchao Chen, Qiwei Zhang, Feng Zhou, Xun Xu, Peng Yang, Xiao Wu, Di Liu, Zuotao |
| author_facet | Zhu, Jie Fan, Zhifang Zhu, Xiaoxie Jiang, Yuchen Wang, Hangyu Han, Xintian Ding, Haoran Wang, Xinmin Zhao, Wenlin Gong, Zhen Yang, Huizhi Chai, Zheng Chen, Zhe Zheng, Yuchao Chen, Qiwei Zhang, Feng Zhou, Xun Xu, Peng Yang, Xiao Wu, Di Liu, Zuotao |
| contents | Recent progress on large language models (LLMs) has spurred interest in scaling up recommendation systems, yet two practical obstacles remain. First, training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands. Second, most human-designed feature-crossing modules in ranking models were inherited from the CPU era and fail to exploit modern GPUs, resulting in low Model Flops Utilization (MFU) and poor scalability. We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture. RankMixer retains the transformer's high parallelism while replacing quadratic self-attention with multi-head token mixing module for higher efficiency. Besides, RankMixer maintains both the modeling for distinct feature subspaces and cross-feature-space interactions with Per-token FFNs. We further extend it to one billion parameters with a Sparse-MoE variant for higher ROI. A dynamic routing strategy is adapted to address the inadequacy and imbalance of experts training. Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset. By replacing previously diverse handcrafted low-MFU modules with RankMixer, we boost the model MFU from 4.5\% to 45\%, and scale our ranking model parameters by 100x while maintaining roughly the same inference latency. We verify RankMixer's universality with online A/B tests across two core application scenarios (Recommendation and Advertisement). Finally, we launch 1B Dense-Parameters RankMixer for full traffic serving without increasing the serving cost, which improves user active days by 0.3\% and total in-app usage duration by 1.08\%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15551 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RankMixer: Scaling Up Ranking Models in Industrial Recommenders Zhu, Jie Fan, Zhifang Zhu, Xiaoxie Jiang, Yuchen Wang, Hangyu Han, Xintian Ding, Haoran Wang, Xinmin Zhao, Wenlin Gong, Zhen Yang, Huizhi Chai, Zheng Chen, Zhe Zheng, Yuchao Chen, Qiwei Zhang, Feng Zhou, Xun Xu, Peng Yang, Xiao Wu, Di Liu, Zuotao Information Retrieval Recent progress on large language models (LLMs) has spurred interest in scaling up recommendation systems, yet two practical obstacles remain. First, training and serving cost on industrial Recommenders must respect strict latency bounds and high QPS demands. Second, most human-designed feature-crossing modules in ranking models were inherited from the CPU era and fail to exploit modern GPUs, resulting in low Model Flops Utilization (MFU) and poor scalability. We introduce RankMixer, a hardware-aware model design tailored towards a unified and scalable feature-interaction architecture. RankMixer retains the transformer's high parallelism while replacing quadratic self-attention with multi-head token mixing module for higher efficiency. Besides, RankMixer maintains both the modeling for distinct feature subspaces and cross-feature-space interactions with Per-token FFNs. We further extend it to one billion parameters with a Sparse-MoE variant for higher ROI. A dynamic routing strategy is adapted to address the inadequacy and imbalance of experts training. Experiments show RankMixer's superior scaling abilities on a trillion-scale production dataset. By replacing previously diverse handcrafted low-MFU modules with RankMixer, we boost the model MFU from 4.5\% to 45\%, and scale our ranking model parameters by 100x while maintaining roughly the same inference latency. We verify RankMixer's universality with online A/B tests across two core application scenarios (Recommendation and Advertisement). Finally, we launch 1B Dense-Parameters RankMixer for full traffic serving without increasing the serving cost, which improves user active days by 0.3\% and total in-app usage duration by 1.08\%. |
| title | RankMixer: Scaling Up Ranking Models in Industrial Recommenders |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2507.15551 |