Mining Area Skyline Objects from Map-based Big Data using Apache Spark Framework
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909160499576832 |
|---|---|
| author | Li, Chen Zhu, Ye Cao, Yang Zhang, Jinli Annisa, Annisa Cheng, Debo Morimoto, Yasuhiko |
| author_facet | Li, Chen Zhu, Ye Cao, Yang Zhang, Jinli Annisa, Annisa Cheng, Debo Morimoto, Yasuhiko |
| contents | The computation of the skyline provides a mechanism for utilizing multiple location-based criteria to identify optimal data points. However, the efficiency of these computations diminishes and becomes more challenging as the input data expands. This study presents a novel algorithm aimed at mitigating this challenge by harnessing the capabilities of Apache Spark, a distributed processing platform, for conducting area skyline computations. The proposed algorithm enhances processing speed and scalability. In particular, our algorithm encompasses three key phases: the computation of distances between data points, the generation of distance tuples, and the execution of the skyline operators. Notably, the second phase employs a local partial skyline extraction technique to minimize the volume of data transmitted from each executor (a parallel processing procedure) to the driver (a central processing procedure). Afterwards, the driver processes the received data to determine the final skyline and creates filters to exclude irrelevant points. Extensive experimentation on eight datasets reveals that our algorithm significantly reduces both data size and computation time required for area skyline computation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03254 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mining Area Skyline Objects from Map-based Big Data using Apache Spark Framework Li, Chen Zhu, Ye Cao, Yang Zhang, Jinli Annisa, Annisa Cheng, Debo Morimoto, Yasuhiko Distributed, Parallel, and Cluster Computing The computation of the skyline provides a mechanism for utilizing multiple location-based criteria to identify optimal data points. However, the efficiency of these computations diminishes and becomes more challenging as the input data expands. This study presents a novel algorithm aimed at mitigating this challenge by harnessing the capabilities of Apache Spark, a distributed processing platform, for conducting area skyline computations. The proposed algorithm enhances processing speed and scalability. In particular, our algorithm encompasses three key phases: the computation of distances between data points, the generation of distance tuples, and the execution of the skyline operators. Notably, the second phase employs a local partial skyline extraction technique to minimize the volume of data transmitted from each executor (a parallel processing procedure) to the driver (a central processing procedure). Afterwards, the driver processes the received data to determine the final skyline and creates filters to exclude irrelevant points. Extensive experimentation on eight datasets reveals that our algorithm significantly reduces both data size and computation time required for area skyline computation. |
| title | Mining Area Skyline Objects from Map-based Big Data using Apache Spark Framework |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2404.03254 |