Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910839552868352 |
|---|---|
| author | Zhang, Yifei Zhu, Hao Yang, Menglin Liu, Jiahong Ying, Rex King, Irwin Koniusz, Piotr |
| author_facet | Zhang, Yifei Zhu, Hao Yang, Menglin Liu, Jiahong Ying, Rex King, Irwin Koniusz, Piotr |
| contents | Learning generalizable self-supervised graph representations for downstream tasks is challenging. To this end, Contrastive Learning (CL) has emerged as a leading approach. The embeddings of CL are arranged on a hypersphere where similarity is measured by the cosine distance. However, many real-world graphs, especially of hierarchical nature, cannot be embedded well in the Euclidean space. Although the hyperbolic embedding is suitable for hierarchical representation learning, naively applying CL to the hyperbolic space may result in the so-called dimension collapse, i.e., features will concentrate mostly within few density regions, leading to poor utilization of the whole feature space. Thus, we propose a novel contrastive learning framework to learn high-quality graph embeddings in hyperbolic space. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information, as well as we propose a substitute of the uniformity metric to prevent the so-called dimensional collapse. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity related to properties of trees. In the ambient space of the hyperbolic manifold these notions translate into imposing an isotropic ring density towards boundaries of Poincaré ball. Our experiments support the efficacy of our method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_18209 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning Zhang, Yifei Zhu, Hao Yang, Menglin Liu, Jiahong Ying, Rex King, Irwin Koniusz, Piotr Machine Learning Artificial Intelligence Learning generalizable self-supervised graph representations for downstream tasks is challenging. To this end, Contrastive Learning (CL) has emerged as a leading approach. The embeddings of CL are arranged on a hypersphere where similarity is measured by the cosine distance. However, many real-world graphs, especially of hierarchical nature, cannot be embedded well in the Euclidean space. Although the hyperbolic embedding is suitable for hierarchical representation learning, naively applying CL to the hyperbolic space may result in the so-called dimension collapse, i.e., features will concentrate mostly within few density regions, leading to poor utilization of the whole feature space. Thus, we propose a novel contrastive learning framework to learn high-quality graph embeddings in hyperbolic space. Specifically, we design the alignment metric that effectively captures the hierarchical data-invariant information, as well as we propose a substitute of the uniformity metric to prevent the so-called dimensional collapse. We show that in the hyperbolic space one has to address the leaf- and height-level uniformity related to properties of trees. In the ambient space of the hyperbolic manifold these notions translate into imposing an isotropic ring density towards boundaries of Poincaré ball. Our experiments support the efficacy of our method. |
| title | Understanding and Mitigating Hyperbolic Dimensional Collapse in Graph Contrastive Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.18209 |