Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.08592 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909752574869504 |
|---|---|
| author | Xu, Liyan Su, Zhenlin Yu, Mo Li, Jiangnan Meng, Fandong Zhou, Jie |
| author_facet | Xu, Liyan Su, Zhenlin Yu, Mo Li, Jiangnan Meng, Fandong Zhou, Jie |
| contents | This work stems from an observed limitation of text encoders: embeddings may not be able to recognize fine-grained entities or events within encoded semantics, resulting in failed retrieval even in simple cases. To examine such behaviors, we first introduce a new evaluation dataset, CapRetrieval, in which passages are image captions and queries are phrases targeting entity or event concepts in diverse forms. Zero-shot evaluation suggests that encoders often struggle with these fine-grained matching, regardless of training sources or model size. Aiming for enhancement, we proceed to finetune encoders with our proposed data generation strategies, enabling a small 0.1B encoder to outperform the state-of-the-art 7B model. Within this process, we further uncover the granularity dilemma, a challenge for embeddings to capture fine-grained salience while aligning with overall semantics. Our dataset, code and models in this work are publicly released at https://github.com/lxucs/CapRetrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08592 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings Xu, Liyan Su, Zhenlin Yu, Mo Li, Jiangnan Meng, Fandong Zhou, Jie Computation and Language Machine Learning This work stems from an observed limitation of text encoders: embeddings may not be able to recognize fine-grained entities or events within encoded semantics, resulting in failed retrieval even in simple cases. To examine such behaviors, we first introduce a new evaluation dataset, CapRetrieval, in which passages are image captions and queries are phrases targeting entity or event concepts in diverse forms. Zero-shot evaluation suggests that encoders often struggle with these fine-grained matching, regardless of training sources or model size. Aiming for enhancement, we proceed to finetune encoders with our proposed data generation strategies, enabling a small 0.1B encoder to outperform the state-of-the-art 7B model. Within this process, we further uncover the granularity dilemma, a challenge for embeddings to capture fine-grained salience while aligning with overall semantics. Our dataset, code and models in this work are publicly released at https://github.com/lxucs/CapRetrieval. |
| title | Dense Retrievers Can Fail on Simple Queries: Revealing The Granularity Dilemma of Embeddings |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2506.08592 |