STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909355714019328 |
|---|---|
| author | Wu, Shirley Zhao, Shiyu Yasunaga, Michihiro Huang, Kexin Cao, Kaidi Huang, Qian Ioannidis, Vassilis N. Subbian, Karthik Zou, James Leskovec, Jure |
| author_facet | Wu, Shirley Zhao, Shiyu Yasunaga, Michihiro Huang, Kexin Cao, Kaidi Huang, Qian Ioannidis, Vassilis N. Subbian, Karthik Zou, James Leskovec, Jure |
| contents | Answering real-world complex queries, such as complex product search, often requires accurate retrieval from semi-structured knowledge bases that involve blend of unstructured (e.g., textual descriptions of products) and structured (e.g., entity relations of products) information. However, many previous works studied textual and relational retrieval tasks as separate topics. To address the gap, we develop STARK, a large-scale Semi-structure retrieval benchmark on Textual and Relational Knowledge Bases. Our benchmark covers three domains: product search, academic paper search, and queries in precision medicine. We design a novel pipeline to synthesize realistic user queries that integrate diverse relational information and complex textual properties, together with their ground-truth answers (items). We conduct rigorous human evaluation to validate the quality of our synthesized queries. We further enhance the benchmark with high-quality human-generated queries to provide an authentic reference. STARK serves as a comprehensive testbed for evaluating the performance of retrieval systems driven by large language models (LLMs). Our experiments suggest that STARK presents significant challenges to the current retrieval and LLM systems, highlighting the need for more capable semi-structured retrieval systems. The benchmark data and code are available on https://github.com/snap-stanford/STaRK. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_13207 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases Wu, Shirley Zhao, Shiyu Yasunaga, Michihiro Huang, Kexin Cao, Kaidi Huang, Qian Ioannidis, Vassilis N. Subbian, Karthik Zou, James Leskovec, Jure Information Retrieval Machine Learning Answering real-world complex queries, such as complex product search, often requires accurate retrieval from semi-structured knowledge bases that involve blend of unstructured (e.g., textual descriptions of products) and structured (e.g., entity relations of products) information. However, many previous works studied textual and relational retrieval tasks as separate topics. To address the gap, we develop STARK, a large-scale Semi-structure retrieval benchmark on Textual and Relational Knowledge Bases. Our benchmark covers three domains: product search, academic paper search, and queries in precision medicine. We design a novel pipeline to synthesize realistic user queries that integrate diverse relational information and complex textual properties, together with their ground-truth answers (items). We conduct rigorous human evaluation to validate the quality of our synthesized queries. We further enhance the benchmark with high-quality human-generated queries to provide an authentic reference. STARK serves as a comprehensive testbed for evaluating the performance of retrieval systems driven by large language models (LLMs). Our experiments suggest that STARK presents significant challenges to the current retrieval and LLM systems, highlighting the need for more capable semi-structured retrieval systems. The benchmark data and code are available on https://github.com/snap-stanford/STaRK. |
| title | STaRK: Benchmarking LLM Retrieval on Textual and Relational Knowledge Bases |
| topic | Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2404.13207 |