TabSketchFM: Sketch-based Tabular Representation Learning for Data Discovery over Data Lakes
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908504833392640 |
|---|---|
| author | Khatiwada, Aamod Kokel, Harsha Abdelaziz, Ibrahim Chaudhury, Subhajit Dolby, Julian Hassanzadeh, Oktie Huang, Zhenhan Pedapati, Tejaswini Samulowitz, Horst Srinivas, Kavitha |
| author_facet | Khatiwada, Aamod Kokel, Harsha Abdelaziz, Ibrahim Chaudhury, Subhajit Dolby, Julian Hassanzadeh, Oktie Huang, Zhenhan Pedapati, Tejaswini Samulowitz, Horst Srinivas, Kavitha |
| contents | Enterprises have a growing need to identify relevant tables in data lakes; e.g. tables that are unionable, joinable, or subsets of each other. Tabular neural models can be helpful for such data discovery tasks. In this paper, we present TabSketchFM, a neural tabular model for data discovery over data lakes. First, we propose novel pre-training: a sketch-based approach to enhance the effectiveness of data discovery in neural tabular models. Second, we finetune the pretrained model for identifying unionable, joinable, and subset table pairs and show significant improvement over previous tabular neural models. Third, we present a detailed ablation study to highlight which sketches are crucial for which tasks. Fourth, we use these finetuned models to perform table search; i.e., given a query table, find other tables in a corpus that are unionable, joinable, or that are subsets of the query. Our results demonstrate significant improvements in F1 scores for search compared to state-of-the-art techniques. Finally, we show significant transfer across datasets and tasks establishing that our model can generalize across different tasks and over different data lakes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01619 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TabSketchFM: Sketch-based Tabular Representation Learning for Data Discovery over Data Lakes Khatiwada, Aamod Kokel, Harsha Abdelaziz, Ibrahim Chaudhury, Subhajit Dolby, Julian Hassanzadeh, Oktie Huang, Zhenhan Pedapati, Tejaswini Samulowitz, Horst Srinivas, Kavitha Machine Learning Artificial Intelligence Databases Enterprises have a growing need to identify relevant tables in data lakes; e.g. tables that are unionable, joinable, or subsets of each other. Tabular neural models can be helpful for such data discovery tasks. In this paper, we present TabSketchFM, a neural tabular model for data discovery over data lakes. First, we propose novel pre-training: a sketch-based approach to enhance the effectiveness of data discovery in neural tabular models. Second, we finetune the pretrained model for identifying unionable, joinable, and subset table pairs and show significant improvement over previous tabular neural models. Third, we present a detailed ablation study to highlight which sketches are crucial for which tasks. Fourth, we use these finetuned models to perform table search; i.e., given a query table, find other tables in a corpus that are unionable, joinable, or that are subsets of the query. Our results demonstrate significant improvements in F1 scores for search compared to state-of-the-art techniques. Finally, we show significant transfer across datasets and tasks establishing that our model can generalize across different tasks and over different data lakes. |
| title | TabSketchFM: Sketch-based Tabular Representation Learning for Data Discovery over Data Lakes |
| topic | Machine Learning Artificial Intelligence Databases |
| url | https://arxiv.org/abs/2407.01619 |