It's Hard to HAC with Average Linkage!
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916597662220288 |
|---|---|
| author | Bateni, MohammadHossein Dhulipala, Laxman Gowda, Kishen N Hershkowitz, D Ellis Jayaram, Rajesh Łącki, Jakub |
| author_facet | Bateni, MohammadHossein Dhulipala, Laxman Gowda, Kishen N Hershkowitz, D Ellis Jayaram, Rajesh Łącki, Jakub |
| contents | Average linkage Hierarchical Agglomerative Clustering (HAC) is an extensively studied and applied method for hierarchical clustering. Recent applications to massive datasets have driven significant interest in near-linear-time and efficient parallel algorithms for average linkage HAC.
We provide hardness results that rule out such algorithms. On the sequential side, we establish a runtime lower bound of $n^{3/2-ε}$ on $n$ node graphs for sequential combinatorial algorithms under standard fine-grained complexity assumptions. This essentially matches the best-known running time for average linkage HAC. On the parallel side, we prove that average linkage HAC likely cannot be parallelized even on simple graphs by showing that it is CC-hard on trees of diameter $4$. On the possibility side, we demonstrate that average linkage HAC can be efficiently parallelized (i.e., it is in NC) on paths and can be solved in near-linear time when the height of the output cluster hierarchy is small. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14730 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | It's Hard to HAC with Average Linkage! Bateni, MohammadHossein Dhulipala, Laxman Gowda, Kishen N Hershkowitz, D Ellis Jayaram, Rajesh Łącki, Jakub Data Structures and Algorithms Computational Complexity Distributed, Parallel, and Cluster Computing Average linkage Hierarchical Agglomerative Clustering (HAC) is an extensively studied and applied method for hierarchical clustering. Recent applications to massive datasets have driven significant interest in near-linear-time and efficient parallel algorithms for average linkage HAC. We provide hardness results that rule out such algorithms. On the sequential side, we establish a runtime lower bound of $n^{3/2-ε}$ on $n$ node graphs for sequential combinatorial algorithms under standard fine-grained complexity assumptions. This essentially matches the best-known running time for average linkage HAC. On the parallel side, we prove that average linkage HAC likely cannot be parallelized even on simple graphs by showing that it is CC-hard on trees of diameter $4$. On the possibility side, we demonstrate that average linkage HAC can be efficiently parallelized (i.e., it is in NC) on paths and can be solved in near-linear time when the height of the output cluster hierarchy is small. |
| title | It's Hard to HAC with Average Linkage! |
| topic | Data Structures and Algorithms Computational Complexity Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2404.14730 |