Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911893052981248 |
|---|---|
| author | Singh, Anima Vu, Trung Mehta, Nikhil Keshavan, Raghunandan Sathiamoorthy, Maheswaran Zheng, Yilin Hong, Lichan Heldt, Lukasz Wei, Li Tandon, Devansh Chi, Ed H. Yi, Xinyang |
| author_facet | Singh, Anima Vu, Trung Mehta, Nikhil Keshavan, Raghunandan Sathiamoorthy, Maheswaran Zheng, Yilin Hong, Lichan Heldt, Lukasz Wei, Li Tandon, Devansh Chi, Ed H. Yi, Xinyang |
| contents | Randomly-hashed item ids are used ubiquitously in recommendation models. However, the learned representations from random hashing prevents generalization across similar items, causing problems of learning unseen and long-tail items, especially when item corpus is large, power-law distributed, and evolving dynamically. In this paper, we propose using content-derived features as a replacement for random ids. We show that simply replacing ID features with content-based embeddings can cause a drop in quality due to reduced memorization capability. To strike a good balance of memorization and generalization, we propose to use Semantic IDs -- a compact discrete item representation learned from frozen content embeddings using RQ-VAE that captures the hierarchy of concepts in items -- as a replacement for random item ids. Similar to content embeddings, the compactness of Semantic IDs poses a problem of easy adaption in recommendation models. We propose novel methods for adapting Semantic IDs in industry-scale ranking models, through hashing sub-pieces of of the Semantic-ID sequences. In particular, we find that the SentencePiece model that is commonly used in LLM tokenization outperforms manually crafted pieces such as N-grams. To the end, we evaluate our approaches in a real-world ranking model for YouTube recommendations. Our experiments demonstrate that Semantic IDs can replace the direct use of video IDs by improving the generalization ability on new and long-tail item slices without sacrificing overall model quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_08121 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations Singh, Anima Vu, Trung Mehta, Nikhil Keshavan, Raghunandan Sathiamoorthy, Maheswaran Zheng, Yilin Hong, Lichan Heldt, Lukasz Wei, Li Tandon, Devansh Chi, Ed H. Yi, Xinyang Information Retrieval Machine Learning Randomly-hashed item ids are used ubiquitously in recommendation models. However, the learned representations from random hashing prevents generalization across similar items, causing problems of learning unseen and long-tail items, especially when item corpus is large, power-law distributed, and evolving dynamically. In this paper, we propose using content-derived features as a replacement for random ids. We show that simply replacing ID features with content-based embeddings can cause a drop in quality due to reduced memorization capability. To strike a good balance of memorization and generalization, we propose to use Semantic IDs -- a compact discrete item representation learned from frozen content embeddings using RQ-VAE that captures the hierarchy of concepts in items -- as a replacement for random item ids. Similar to content embeddings, the compactness of Semantic IDs poses a problem of easy adaption in recommendation models. We propose novel methods for adapting Semantic IDs in industry-scale ranking models, through hashing sub-pieces of of the Semantic-ID sequences. In particular, we find that the SentencePiece model that is commonly used in LLM tokenization outperforms manually crafted pieces such as N-grams. To the end, we evaluate our approaches in a real-world ranking model for YouTube recommendations. Our experiments demonstrate that Semantic IDs can replace the direct use of video IDs by improving the generalization ability on new and long-tail item slices without sacrificing overall model quality. |
| title | Better Generalization with Semantic IDs: A Case Study in Ranking for Recommendations |
| topic | Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2306.08121 |