FREYJA: Efficient Join Discovery in Data Lakes
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915747658203136 |
|---|---|
| author | Maynou, Marc Nadal, Sergi Panadero, Raquel Flores, Javier Romero, Oscar Queralt, Anna |
| author_facet | Maynou, Marc Nadal, Sergi Panadero, Raquel Flores, Javier Romero, Oscar Queralt, Anna |
| contents | Data lakes are massive repositories of raw and heterogeneous data, designed to meet the requirements of modern data storage. Nonetheless, this same philosophy increases the complexity of performing discovery tasks to find relevant data for subsequent processing. As a response to these growing challenges, we present FREYJA, a modern data discovery system capable of effectively exploring data lakes, aimed at finding candidates to perform joins and increase the number of attributes for downstream tasks. More precisely, we want to compute rankings that sort potential joins by their relevance. Modern mechanisms apply advanced table representation learning (TRL) techniques to yield accurate joins. Yet, this incurs high computational costs when dealing with elevated volumes of data. In contrast to the state-of-the-art, we adopt a novel notion of join quality tailored to data lakes, which leverages syntactic measurements while achieving accuracy comparable to that of TRL approaches. To obtain this metric in a scalable manner we train a general purpose predictive model. Predictions are based, rather than on large-scale datasets, on data profiles, succinct representations that capture the underlying characteristics of the data. Our experiments show that our system, FREYJA, matches the results of the state-of-the-art whilst reducing the execution times by several orders of magnitude. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_06637 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FREYJA: Efficient Join Discovery in Data Lakes Maynou, Marc Nadal, Sergi Panadero, Raquel Flores, Javier Romero, Oscar Queralt, Anna Databases Data lakes are massive repositories of raw and heterogeneous data, designed to meet the requirements of modern data storage. Nonetheless, this same philosophy increases the complexity of performing discovery tasks to find relevant data for subsequent processing. As a response to these growing challenges, we present FREYJA, a modern data discovery system capable of effectively exploring data lakes, aimed at finding candidates to perform joins and increase the number of attributes for downstream tasks. More precisely, we want to compute rankings that sort potential joins by their relevance. Modern mechanisms apply advanced table representation learning (TRL) techniques to yield accurate joins. Yet, this incurs high computational costs when dealing with elevated volumes of data. In contrast to the state-of-the-art, we adopt a novel notion of join quality tailored to data lakes, which leverages syntactic measurements while achieving accuracy comparable to that of TRL approaches. To obtain this metric in a scalable manner we train a general purpose predictive model. Predictions are based, rather than on large-scale datasets, on data profiles, succinct representations that capture the underlying characteristics of the data. Our experiments show that our system, FREYJA, matches the results of the state-of-the-art whilst reducing the execution times by several orders of magnitude. |
| title | FREYJA: Efficient Join Discovery in Data Lakes |
| topic | Databases |
| url | https://arxiv.org/abs/2412.06637 |