Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.19349 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918158378467328 |
|---|---|
| author | Zhang, Han Jiang, Yunjiang Li, Mingming Yuan, Haowei Qiu, Yiming Yang, Wen-Yun |
| author_facet | Zhang, Han Jiang, Yunjiang Li, Mingming Yuan, Haowei Qiu, Yiming Yang, Wen-Yun |
| contents | Embedding-based retrieval aims to learn a shared semantic representation space for both queries and items, enabling efficient and effective item retrieval through approximate nearest neighbor (ANN) algorithms. In current industrial practice, retrieval systems typically retrieve a fixed number of items for each query. However, this fixed-size retrieval often results in insufficient recall for head queries and low precision for tail queries. This limitation largely stems from the dominance of frequentist approaches in loss function design, which fail to address this challenge in industry. In this paper, we propose a novel \textbf{p}robabilistic \textbf{E}mbedding-\textbf{B}ased \textbf{R}etrieval (\textbf{pEBR}) framework. Our method models the item distribution conditioned on each query, enabling the use of a dynamic cosine similarity threshold derived from the cumulative distribution function (CDF) of the probabilistic model. Experimental results demonstrate that pEBR significantly improves both retrieval precision and recall. Furthermore, ablation studies reveal that the probabilistic formulation effectively captures the inherent differences between head-to-tail queries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_19349 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | pEBR: A Probabilistic Approach to Embedding Based Retrieval Zhang, Han Jiang, Yunjiang Li, Mingming Yuan, Haowei Qiu, Yiming Yang, Wen-Yun Information Retrieval Artificial Intelligence Embedding-based retrieval aims to learn a shared semantic representation space for both queries and items, enabling efficient and effective item retrieval through approximate nearest neighbor (ANN) algorithms. In current industrial practice, retrieval systems typically retrieve a fixed number of items for each query. However, this fixed-size retrieval often results in insufficient recall for head queries and low precision for tail queries. This limitation largely stems from the dominance of frequentist approaches in loss function design, which fail to address this challenge in industry. In this paper, we propose a novel \textbf{p}robabilistic \textbf{E}mbedding-\textbf{B}ased \textbf{R}etrieval (\textbf{pEBR}) framework. Our method models the item distribution conditioned on each query, enabling the use of a dynamic cosine similarity threshold derived from the cumulative distribution function (CDF) of the probabilistic model. Experimental results demonstrate that pEBR significantly improves both retrieval precision and recall. Furthermore, ablation studies reveal that the probabilistic formulation effectively captures the inherent differences between head-to-tail queries. |
| title | pEBR: A Probabilistic Approach to Embedding Based Retrieval |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2410.19349 |