Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915212963086336 |
|---|---|
| author | CH-Wang, Sky Deshpande, Darshan Muresan, Smaranda Kannappan, Anand Qian, Rebecca |
| author_facet | CH-Wang, Sky Deshpande, Darshan Muresan, Smaranda Kannappan, Anand Qian, Rebecca |
| contents | We introduce Browsing Lost Unformed Recollections, a tip-of-the-tongue known-item search and reasoning benchmark for general AI assistants. BLUR introduces a set of 573 real-world validated questions that demand searching and reasoning across multi-modal and multilingual inputs, as well as proficient tool use, in order to excel on. Humans easily ace these questions (scoring on average 98%), while the best-performing system scores around 56%. To facilitate progress toward addressing this challenging and aspirational use case for general AI assistants, we release 350 questions through a public leaderboard, retain the answers to 250 of them, and have the rest as a private test set. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_19193 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning CH-Wang, Sky Deshpande, Darshan Muresan, Smaranda Kannappan, Anand Qian, Rebecca Artificial Intelligence Computation and Language Information Retrieval Multiagent Systems We introduce Browsing Lost Unformed Recollections, a tip-of-the-tongue known-item search and reasoning benchmark for general AI assistants. BLUR introduces a set of 573 real-world validated questions that demand searching and reasoning across multi-modal and multilingual inputs, as well as proficient tool use, in order to excel on. Humans easily ace these questions (scoring on average 98%), while the best-performing system scores around 56%. To facilitate progress toward addressing this challenging and aspirational use case for general AI assistants, we release 350 questions through a public leaderboard, retain the answers to 250 of them, and have the rest as a private test set. |
| title | Browsing Lost Unformed Recollections: A Benchmark for Tip-of-the-Tongue Search and Reasoning |
| topic | Artificial Intelligence Computation and Language Information Retrieval Multiagent Systems |
| url | https://arxiv.org/abs/2503.19193 |