Leveraging Task Structures for Improved Identifiability in Neural Network Representations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929470328274944 |
|---|---|
| author | Chen, Wenlin Horwood, Julien Heo, Juyeon Hernández-Lobato, José Miguel |
| author_facet | Chen, Wenlin Horwood, Julien Heo, Juyeon Hernández-Lobato, José Miguel |
| contents | This work extends the theory of identifiability in supervised learning by considering the consequences of having access to a distribution of tasks. In such cases, we show that linear identifiability is achievable in the general multi-task regression setting. Furthermore, we show that the existence of a task distribution which defines a conditional prior over latent factors reduces the equivalence class for identifiability to permutations and scaling of the true latent factors, a stronger and more useful result than linear identifiability. Crucially, when we further assume a causal structure over these tasks, our approach enables simple maximum marginal likelihood optimization, and suggests potential downstream applications to causal representation learning. Empirically, we find that this straightforward optimization procedure enables our model to outperform more general unsupervised models in recovering canonical representations for both synthetic data and real-world molecular data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_14861 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Leveraging Task Structures for Improved Identifiability in Neural Network Representations Chen, Wenlin Horwood, Julien Heo, Juyeon Hernández-Lobato, José Miguel Machine Learning This work extends the theory of identifiability in supervised learning by considering the consequences of having access to a distribution of tasks. In such cases, we show that linear identifiability is achievable in the general multi-task regression setting. Furthermore, we show that the existence of a task distribution which defines a conditional prior over latent factors reduces the equivalence class for identifiability to permutations and scaling of the true latent factors, a stronger and more useful result than linear identifiability. Crucially, when we further assume a causal structure over these tasks, our approach enables simple maximum marginal likelihood optimization, and suggests potential downstream applications to causal representation learning. Empirically, we find that this straightforward optimization procedure enables our model to outperform more general unsupervised models in recovering canonical representations for both synthetic data and real-world molecular data. |
| title | Leveraging Task Structures for Improved Identifiability in Neural Network Representations |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2306.14861 |