Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.06657 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909830996819968 |
|---|---|
| author | Long, Boyuan Wang, Yueqi Mehta, Hiloni Zomnir, Mick Pathak, Omkar Meng, Changping Jia, Ruolin Peng, Yajun Hong, Dapeng Wu, Xia Gao, Mingyan Dalal, Onkar Han, Ningren |
| author_facet | Long, Boyuan Wang, Yueqi Mehta, Hiloni Zomnir, Mick Pathak, Omkar Meng, Changping Jia, Ruolin Peng, Yajun Hong, Dapeng Wu, Xia Gao, Mingyan Dalal, Onkar Han, Ningren |
| contents | This paper presents a case study on deploying Large Language Models (LLMs) as an advanced "annotation" mechanism to achieve nuanced content understanding (e.g., discerning content "vibe") at scale within a large-scale industrial short-form video recommendation system. Traditional machine learning classifiers for content understanding face protracted development cycles and a lack of deep, nuanced comprehension. The "LLM-as-annotators" approach addresses these by significantly shortening development times and enabling the annotation of subtle attributes. This work details an end-to-end workflow encompassing: (1) iterative definition and robust evaluation of target attributes, refined by offline metrics and online A/B testing; (2) scalable offline bulk annotation of video corpora using LLMs with multimodal features, optimized inference, and knowledge distillation for broad application; and (3) integration of these rich annotations into the online recommendation serving system, for example, through personalized restrict retrieval. Experimental results demonstrate the efficacy of this approach, with LLMs outperforming human raters in offline annotation quality for nuanced attributes and yielding significant improvements of user participation and satisfied consumption in online A/B tests. The study provides insights into designing and scaling production-level LLM pipelines for rich content evaluation, highlighting the adaptability and benefits of LLM-generated nuanced understanding for enhancing content discovery, user satisfaction, and the overall effectiveness of modern recommendation systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06657 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LLM-Powered Nuanced Video Attribute Annotation for Enhanced Recommendations Long, Boyuan Wang, Yueqi Mehta, Hiloni Zomnir, Mick Pathak, Omkar Meng, Changping Jia, Ruolin Peng, Yajun Hong, Dapeng Wu, Xia Gao, Mingyan Dalal, Onkar Han, Ningren Information Retrieval This paper presents a case study on deploying Large Language Models (LLMs) as an advanced "annotation" mechanism to achieve nuanced content understanding (e.g., discerning content "vibe") at scale within a large-scale industrial short-form video recommendation system. Traditional machine learning classifiers for content understanding face protracted development cycles and a lack of deep, nuanced comprehension. The "LLM-as-annotators" approach addresses these by significantly shortening development times and enabling the annotation of subtle attributes. This work details an end-to-end workflow encompassing: (1) iterative definition and robust evaluation of target attributes, refined by offline metrics and online A/B testing; (2) scalable offline bulk annotation of video corpora using LLMs with multimodal features, optimized inference, and knowledge distillation for broad application; and (3) integration of these rich annotations into the online recommendation serving system, for example, through personalized restrict retrieval. Experimental results demonstrate the efficacy of this approach, with LLMs outperforming human raters in offline annotation quality for nuanced attributes and yielding significant improvements of user participation and satisfied consumption in online A/B tests. The study provides insights into designing and scaling production-level LLM pipelines for rich content evaluation, highlighting the adaptability and benefits of LLM-generated nuanced understanding for enhancing content discovery, user satisfaction, and the overall effectiveness of modern recommendation systems. |
| title | LLM-Powered Nuanced Video Attribute Annotation for Enhanced Recommendations |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2510.06657 |