RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929599455166464 |
|---|---|
| author | Choi, Nayoung Lee, Youngjune Cho, Gyu-Hwung Jeong, Haeyu Kong, Jungmin Kim, Saehun Park, Keunchan Cho, Sarah Jeong, Inchang Nam, Gyohee Han, Sunghoon Yang, Wonil Choi, Jaeho |
| author_facet | Choi, Nayoung Lee, Youngjune Cho, Gyu-Hwung Jeong, Haeyu Kong, Jungmin Kim, Saehun Park, Keunchan Cho, Sarah Jeong, Inchang Nam, Gyohee Han, Sunghoon Yang, Wonil Choi, Jaeho |
| contents | Large Language Models (LLMs) excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. These queries are challenging for feedback-based rankings due to sparse user engagement and limited feedback, making LLMs' ranking ability highly valuable. However, the large size and slow inference of LLMs necessitate the development of smaller, more efficient models (sLLMs). Recently, integrating ranking label generation into distillation techniques has become crucial, but existing methods underutilize LLMs' capabilities and are cumbersome. Our research, RRADistill: Re-Ranking Ability Distillation, propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. We introduce an encoder-based method using a Term Control Layer to capture term matching signals and a decoder-based model with a ranking layer for enhanced understanding. A/B testing on a Korean-based search platform, validates the effectiveness of our approach in improving re-ranking for long-tail queries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18097 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine Choi, Nayoung Lee, Youngjune Cho, Gyu-Hwung Jeong, Haeyu Kong, Jungmin Kim, Saehun Park, Keunchan Cho, Sarah Jeong, Inchang Nam, Gyohee Han, Sunghoon Yang, Wonil Choi, Jaeho Information Retrieval Artificial Intelligence Machine Learning Large Language Models (LLMs) excel at understanding the semantic relationships between queries and documents, even with lengthy and complex long-tail queries. These queries are challenging for feedback-based rankings due to sparse user engagement and limited feedback, making LLMs' ranking ability highly valuable. However, the large size and slow inference of LLMs necessitate the development of smaller, more efficient models (sLLMs). Recently, integrating ranking label generation into distillation techniques has become crucial, but existing methods underutilize LLMs' capabilities and are cumbersome. Our research, RRADistill: Re-Ranking Ability Distillation, propose an efficient label generation pipeline and novel sLLM training methods for both encoder and decoder models. We introduce an encoder-based method using a Term Control Layer to capture term matching signals and a decoder-based model with a ranking layer for enhanced understanding. A/B testing on a Korean-based search platform, validates the effectiveness of our approach in improving re-ranking for long-tail queries. |
| title | RRADistill: Distilling LLMs' Passage Ranking Ability for Long-Tail Queries Document Re-Ranking on a Search Engine |
| topic | Information Retrieval Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.18097 |