Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912931060383744 |
|---|---|
| author | Joshi, Shruti Saulus, Théo Brendel, Wieland Brouillard, Philippe Sridhar, Dhanya Reizinger, Patrik |
| author_facet | Joshi, Shruti Saulus, Théo Brendel, Wieland Brouillard, Philippe Sridhar, Dhanya Reizinger, Patrik |
| contents | Identifiability in representation learning is commonly evaluated using standard metrics (e.g., MCC, DCI, R^2) on synthetic benchmarks with known ground-truth factors. These metrics are assumed to reflect recovery up to the equivalence class guaranteed by identifiability theory. We show that this assumption holds only under specific structural conditions: each metric implicitly encodes assumptions about both the data-generating process (DGP) and the encoder. When these assumptions are violated, metrics become misspecified and can produce systematic false positives and false negatives. Such failures occur both within classical identifiability regimes and in post-hoc settings where identifiability is most needed. We introduce a taxonomy separating DGP assumptions from encoder geometry, use it to characterise the validity domains of existing metrics, and release an evaluation suite for reproducible stress testing and comparison. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_24278 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations Joshi, Shruti Saulus, Théo Brendel, Wieland Brouillard, Philippe Sridhar, Dhanya Reizinger, Patrik Machine Learning Identifiability in representation learning is commonly evaluated using standard metrics (e.g., MCC, DCI, R^2) on synthetic benchmarks with known ground-truth factors. These metrics are assumed to reflect recovery up to the equivalence class guaranteed by identifiability theory. We show that this assumption holds only under specific structural conditions: each metric implicitly encodes assumptions about both the data-generating process (DGP) and the encoder. When these assumptions are violated, metrics become misspecified and can produce systematic false positives and false negatives. Such failures occur both within classical identifiability regimes and in post-hoc settings where identifiability is most needed. We introduce a taxonomy separating DGP assumptions from encoder geometry, use it to characterise the validity domains of existing metrics, and release an evaluation suite for reproducible stress testing and comparison. |
| title | Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representations |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.24278 |