Semantic Search Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913566091640832 |
|---|---|
| author | Zheng, Chujie Wang, Jeffrey Zhang, Shuqian Albee Kishore, Anand Singh, Siddharth |
| author_facet | Zheng, Chujie Wang, Jeffrey Zhang, Shuqian Albee Kishore, Anand Singh, Siddharth |
| contents | We propose a novel method for evaluating the performance of a content search system that measures the semantic match between a query and the results returned by the search system. We introduce a metric called "on-topic rate" to measure the percentage of results that are relevant to the query. To achieve this, we design a pipeline that defines a golden query set, retrieves the top K results for each query, and sends calls to GPT 3.5 with formulated prompts. Our semantic evaluation pipeline helps identify common failure patterns and goals against the metric for relevance improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_21549 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Semantic Search Evaluation Zheng, Chujie Wang, Jeffrey Zhang, Shuqian Albee Kishore, Anand Singh, Siddharth Information Retrieval Computation and Language We propose a novel method for evaluating the performance of a content search system that measures the semantic match between a query and the results returned by the search system. We introduce a metric called "on-topic rate" to measure the percentage of results that are relevant to the query. To achieve this, we design a pipeline that defines a golden query set, retrieves the top K results for each query, and sends calls to GPT 3.5 with formulated prompts. Our semantic evaluation pipeline helps identify common failure patterns and goals against the metric for relevance improvements. |
| title | Semantic Search Evaluation |
| topic | Information Retrieval Computation and Language |
| url | https://arxiv.org/abs/2410.21549 |