Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917657924599808 |
|---|---|
| author | Zhai, Jiaqi Liao, Lucy Liu, Xing Wang, Yueming Li, Rui Cao, Xuan Gao, Leon Gong, Zhaojie Gu, Fangda He, Michael Lu, Yinghai Shi, Yu |
| author_facet | Zhai, Jiaqi Liao, Lucy Liu, Xing Wang, Yueming Li, Rui Cao, Xuan Gao, Leon Gong, Zhaojie Gu, Fangda He, Michael Lu, Yinghai Shi, Yu |
| contents | Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute.
Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data.
HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17152 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations Zhai, Jiaqi Liao, Lucy Liu, Xing Wang, Yueming Li, Rui Cao, Xuan Gao, Leon Gong, Zhaojie Gu, Fangda He, Michael Lu, Yinghai Shi, Yu Machine Learning Information Retrieval Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute. Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data. HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations. |
| title | Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations |
| topic | Machine Learning Information Retrieval |
| url | https://arxiv.org/abs/2402.17152 |