Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.00519 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917892739563520 |
|---|---|
| author | Kong, Lingjing Chen, Guangyi Huang, Biwei Xing, Eric P. Chi, Yuejie Zhang, Kun |
| author_facet | Kong, Lingjing Chen, Guangyi Huang, Biwei Xing, Eric P. Chi, Yuejie Zhang, Kun |
| contents | Learning concepts from natural high-dimensional data (e.g., images) holds potential in building human-aligned and interpretable machine learning models. Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking. In this work, we formalize concepts as discrete latent causal variables that are related via a hierarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e.g., a dog breed and its eye shapes in natural images). We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible. Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyclic graphs in prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images. We substantiate our theoretical claims with synthetic data experiments. Further, we discuss our theory's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_00519 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Learning Discrete Concepts in Latent Hierarchical Models Kong, Lingjing Chen, Guangyi Huang, Biwei Xing, Eric P. Chi, Yuejie Zhang, Kun Machine Learning Artificial Intelligence Learning concepts from natural high-dimensional data (e.g., images) holds potential in building human-aligned and interpretable machine learning models. Despite its encouraging prospect, formalization and theoretical insights into this crucial task are still lacking. In this work, we formalize concepts as discrete latent causal variables that are related via a hierarchical causal model that encodes different abstraction levels of concepts embedded in high-dimensional data (e.g., a dog breed and its eye shapes in natural images). We formulate conditions to facilitate the identification of the proposed causal model, which reveals when learning such concepts from unsupervised data is possible. Our conditions permit complex causal hierarchical structures beyond latent trees and multi-level directed acyclic graphs in prior work and can handle high-dimensional, continuous observed variables, which is well-suited for unstructured data modalities such as images. We substantiate our theoretical claims with synthetic data experiments. Further, we discuss our theory's implications for understanding the underlying mechanisms of latent diffusion models and provide corresponding empirical evidence for our theoretical insights. |
| title | Learning Discrete Concepts in Latent Hierarchical Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2406.00519 |