Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.10432 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908588862078976 |
|---|---|
| author | Zeng, Zhichen Hang, Mengyue Liu, Xiaolong Liu, Xiaoyi Lin, Xiao Qiu, Ruizhong Wei, Tianxin Liu, Zhining Yuan, Siyang Yang, Chaofei Liu, Yiqun Yin, Hang Yang, Jiyan Tong, Hanghang |
| author_facet | Zeng, Zhichen Hang, Mengyue Liu, Xiaolong Liu, Xiaoyi Lin, Xiao Qiu, Ruizhong Wei, Tianxin Liu, Zhining Yuan, Siyang Yang, Chaofei Liu, Yiqun Yin, Hang Yang, Jiyan Tong, Hanghang |
| contents | Deep models have driven significant advances in click-through rate (CTR) prediction. While vertical scaling via layer stacking improves model expressiveness, the layer-by-layer sequential computation poses challenges to efficient scaling. Conversely, horizontal scaling through Mixture of Experts (MoE) achieves efficient scaling by activating a small subset of experts in parallel, but flat MoE layers may struggle to capture the hierarchical structure inherent in recommendation tasks. To push the Return-On-Investment (ROI) boundary, we explore the complementary strengths of both directions and propose HiLoMoE, a hierarchical LoRA MoE framework that enables holistic scaling in a parameter-efficient manner. Specifically, HiLoMoE employs lightweight rank-1 experts for parameter-efficient horizontal scaling, and stacks multiple MoE layers with hierarchical routing to enable combinatorially diverse expert compositions. Unlike conventional stacking, HiLoMoE routes based on prior layer scores rather than outputs, allowing all layers to execute in parallel. A principled three-stage training framework ensures stable optimization and expert diversity. Experiments on four public datasets show that HiLoMoE achieving better performance-efficiency tradeoff, achieving an average AUC improvement of 0.20\% in AUC and 18.5\% reduction in FLOPs compared to the non-MoE baseline. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10432 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hierarchical LoRA MoE for Efficient CTR Model Scaling Zeng, Zhichen Hang, Mengyue Liu, Xiaolong Liu, Xiaoyi Lin, Xiao Qiu, Ruizhong Wei, Tianxin Liu, Zhining Yuan, Siyang Yang, Chaofei Liu, Yiqun Yin, Hang Yang, Jiyan Tong, Hanghang Machine Learning Artificial Intelligence Information Retrieval Deep models have driven significant advances in click-through rate (CTR) prediction. While vertical scaling via layer stacking improves model expressiveness, the layer-by-layer sequential computation poses challenges to efficient scaling. Conversely, horizontal scaling through Mixture of Experts (MoE) achieves efficient scaling by activating a small subset of experts in parallel, but flat MoE layers may struggle to capture the hierarchical structure inherent in recommendation tasks. To push the Return-On-Investment (ROI) boundary, we explore the complementary strengths of both directions and propose HiLoMoE, a hierarchical LoRA MoE framework that enables holistic scaling in a parameter-efficient manner. Specifically, HiLoMoE employs lightweight rank-1 experts for parameter-efficient horizontal scaling, and stacks multiple MoE layers with hierarchical routing to enable combinatorially diverse expert compositions. Unlike conventional stacking, HiLoMoE routes based on prior layer scores rather than outputs, allowing all layers to execute in parallel. A principled three-stage training framework ensures stable optimization and expert diversity. Experiments on four public datasets show that HiLoMoE achieving better performance-efficiency tradeoff, achieving an average AUC improvement of 0.20\% in AUC and 18.5\% reduction in FLOPs compared to the non-MoE baseline. |
| title | Hierarchical LoRA MoE for Efficient CTR Model Scaling |
| topic | Machine Learning Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2510.10432 |