UQE: A Query Engine for Unstructured Databases
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910701980745728 |
|---|---|
| author | Dai, Hanjun Wang, Bethany Yixin Wan, Xingchen Dai, Bo Yang, Sherry Nova, Azade Yin, Pengcheng Phothilimthana, Phitchaya Mangpo Sutton, Charles Schuurmans, Dale |
| author_facet | Dai, Hanjun Wang, Bethany Yixin Wan, Xingchen Dai, Bo Yang, Sherry Nova, Azade Yin, Pengcheng Phothilimthana, Phitchaya Mangpo Sutton, Charles Schuurmans, Dale |
| contents | Analytics on structured data is a mature field with many successful methods. However, most real world data exists in unstructured form, such as images and conversations. We investigate the potential of Large Language Models (LLMs) to enable unstructured data analytics. In particular, we propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections. This engine accepts queries in a Universal Query Language (UQL), a dialect of SQL that provides full natural language flexibility in specifying conditions and operators. The new engine leverages the ability of LLMs to conduct analysis of unstructured data, while also allowing us to exploit advances in sampling and optimization techniques to achieve efficient and accurate query execution. In addition, we borrow techniques from classical compiler theory to better orchestrate the workflow between sampling methods and foundation model calls. We demonstrate the efficiency of UQE on data analytics across different modalities, including images, dialogs and reviews, across a range of useful query types, including conditional aggregation, semantic retrieval and abstraction aggregation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09522 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | UQE: A Query Engine for Unstructured Databases Dai, Hanjun Wang, Bethany Yixin Wan, Xingchen Dai, Bo Yang, Sherry Nova, Azade Yin, Pengcheng Phothilimthana, Phitchaya Mangpo Sutton, Charles Schuurmans, Dale Databases Artificial Intelligence Machine Learning Analytics on structured data is a mature field with many successful methods. However, most real world data exists in unstructured form, such as images and conversations. We investigate the potential of Large Language Models (LLMs) to enable unstructured data analytics. In particular, we propose a new Universal Query Engine (UQE) that directly interrogates and draws insights from unstructured data collections. This engine accepts queries in a Universal Query Language (UQL), a dialect of SQL that provides full natural language flexibility in specifying conditions and operators. The new engine leverages the ability of LLMs to conduct analysis of unstructured data, while also allowing us to exploit advances in sampling and optimization techniques to achieve efficient and accurate query execution. In addition, we borrow techniques from classical compiler theory to better orchestrate the workflow between sampling methods and foundation model calls. We demonstrate the efficiency of UQE on data analytics across different modalities, including images, dialogs and reviews, across a range of useful query types, including conditional aggregation, semantic retrieval and abstraction aggregation. |
| title | UQE: A Query Engine for Unstructured Databases |
| topic | Databases Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2407.09522 |