AdaNDV: Adaptive Number of Distinct Value Estimation via Learning to Select and Fuse Estimators
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910853465374720 |
|---|---|
| author | Xu, Xianghong Zhang, Tieying He, Xiao Li, Haoyang Kang, Rong Wang, Shuai Xu, Linhui Liang, Zhimin Luo, Shangyu Zhang, Lei Chen, Jianjun |
| author_facet | Xu, Xianghong Zhang, Tieying He, Xiao Li, Haoyang Kang, Rong Wang, Shuai Xu, Linhui Liang, Zhimin Luo, Shangyu Zhang, Lei Chen, Jianjun |
| contents | Estimating the Number of Distinct Values (NDV) is fundamental for numerous data management tasks, especially within database applications. However, most existing works primarily focus on introducing new statistical or learned estimators, while identifying the most suitable estimator for a given scenario remains largely unexplored. Therefore, we propose AdaNDV, a learned method designed to adaptively select and fuse existing estimators to address this issue. Specifically, (1) we propose to use learned models to distinguish between overestimated and underestimated estimators and then select appropriate estimators from each category. This strategy provides a complementary perspective by integrating overestimations and underestimations for error correction, thereby improving the accuracy of NDV estimation. (2) To further integrate the estimation results, we introduce a novel fusion approach that employs a learned model to predict the weights of the selected estimators and then applies a weighted sum to merge them. By combining these strategies, the proposed AdaNDV fundamentally distinguishes itself from previous works that directly estimate NDV. Moreover, extensive experiments conducted on real-world datasets, with the number of individual columns being several orders of magnitude larger than in previous studies, demonstrate the superior performance of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16190 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AdaNDV: Adaptive Number of Distinct Value Estimation via Learning to Select and Fuse Estimators Xu, Xianghong Zhang, Tieying He, Xiao Li, Haoyang Kang, Rong Wang, Shuai Xu, Linhui Liang, Zhimin Luo, Shangyu Zhang, Lei Chen, Jianjun Databases Estimating the Number of Distinct Values (NDV) is fundamental for numerous data management tasks, especially within database applications. However, most existing works primarily focus on introducing new statistical or learned estimators, while identifying the most suitable estimator for a given scenario remains largely unexplored. Therefore, we propose AdaNDV, a learned method designed to adaptively select and fuse existing estimators to address this issue. Specifically, (1) we propose to use learned models to distinguish between overestimated and underestimated estimators and then select appropriate estimators from each category. This strategy provides a complementary perspective by integrating overestimations and underestimations for error correction, thereby improving the accuracy of NDV estimation. (2) To further integrate the estimation results, we introduce a novel fusion approach that employs a learned model to predict the weights of the selected estimators and then applies a weighted sum to merge them. By combining these strategies, the proposed AdaNDV fundamentally distinguishes itself from previous works that directly estimate NDV. Moreover, extensive experiments conducted on real-world datasets, with the number of individual columns being several orders of magnitude larger than in previous studies, demonstrate the superior performance of our method. |
| title | AdaNDV: Adaptive Number of Distinct Value Estimation via Learning to Select and Fuse Estimators |
| topic | Databases |
| url | https://arxiv.org/abs/2502.16190 |