Image de couverture de livre

MERGE: Next-Generation Item Indexing Paradigm for Large-Scale Streaming Recommendation

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Yan, Jing, Bai, Yimeng, Liu, Zongyu, Liu, Yahui, Wang, Junwei, Huang, Jingze, Li, Haoda, Ding, Sihao, Ruan, Shaohui, Zhang, Yang
Format:	Preprint
Publié:	2026
Sujets:	Information Retrieval
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

_version_	1866912854177742848
author	Yan, Jing Bai, Yimeng Liu, Zongyu Liu, Yahui Wang, Junwei Huang, Jingze Li, Haoda Ding, Sihao Ruan, Shaohui Zhang, Yang
author_facet	Yan, Jing Bai, Yimeng Liu, Zongyu Liu, Yahui Wang, Junwei Huang, Jingze Li, Haoda Ding, Sihao Ruan, Shaohui Zhang, Yang
contents	Item indexing, which maps a large corpus of items into compact discrete representations, is critical for both discriminative and generative recommender systems, yet existing Vector Quantization (VQ)-based approaches struggle with the highly skewed and non-stationary item distributions common in streaming industry recommenders, leading to poor assignment accuracy, imbalanced cluster occupancy, and insufficient cluster separation. To address these challenges, we propose MERGE, a next-generation item indexing paradigm that adaptively constructs clusters from scratch, dynamically monitors cluster occupancy, and forms hierarchical index structures via fine-to-coarse merging. Extensive experiments demonstrate that MERGE significantly improves assignment accuracy, cluster uniformity, and cluster separation compared with existing indexing methods, while online A/B tests show substantial gains in key business metrics, highlighting its potential as a foundational indexing approach for large-scale recommendation.
format	Preprint
id	arxiv_https___arxiv_org_abs_2601_20199
institution	arXiv
publishDate	2026
record_format	arxiv
spellingShingle	MERGE: Next-Generation Item Indexing Paradigm for Large-Scale Streaming Recommendation Yan, Jing Bai, Yimeng Liu, Zongyu Liu, Yahui Wang, Junwei Huang, Jingze Li, Haoda Ding, Sihao Ruan, Shaohui Zhang, Yang Information Retrieval Item indexing, which maps a large corpus of items into compact discrete representations, is critical for both discriminative and generative recommender systems, yet existing Vector Quantization (VQ)-based approaches struggle with the highly skewed and non-stationary item distributions common in streaming industry recommenders, leading to poor assignment accuracy, imbalanced cluster occupancy, and insufficient cluster separation. To address these challenges, we propose MERGE, a next-generation item indexing paradigm that adaptively constructs clusters from scratch, dynamically monitors cluster occupancy, and forms hierarchical index structures via fine-to-coarse merging. Extensive experiments demonstrate that MERGE significantly improves assignment accuracy, cluster uniformity, and cluster separation compared with existing indexing methods, while online A/B tests show substantial gains in key business metrics, highlighting its potential as a foundational indexing approach for large-scale recommendation.
title	MERGE: Next-Generation Item Indexing Paradigm for Large-Scale Streaming Recommendation
topic	Information Retrieval
url	https://arxiv.org/abs/2601.20199