Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.00607 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915972540006400 |
|---|---|
| author | Shen, Gaofei Bentum, Martijn Lentz, Tom Alishahi, Afra Chrupała, Grzegorz |
| author_facet | Shen, Gaofei Bentum, Martijn Lentz, Tom Alishahi, Afra Chrupała, Grzegorz |
| contents | Probing is widely used to study which features can be decoded from language model representations. However, the common decoding probe approach has two limitations that we aim to solve with our new encoding probe approach: contributions of different features to model representations cannot be directly compared, and feature correlations can affect probing results. We present an Encoding Probe that reverses this direction and reconstructs internal representations of models using interpretable features. We evaluate this method on text and speech transformer models, using feature sets spanning acoustics, phonetics, syntax, lexicon, and speaker identity. Our results suggest that speaker-related effects vary strongly across different training objectives and datasets, while syntactic and lexical features contribute independently to reconstruction. These results show that the Encoding Probe provides a complementary perspective on interpreting model representations beyond decodability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_00607 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe Shen, Gaofei Bentum, Martijn Lentz, Tom Alishahi, Afra Chrupała, Grzegorz Computation and Language Audio and Speech Processing Probing is widely used to study which features can be decoded from language model representations. However, the common decoding probe approach has two limitations that we aim to solve with our new encoding probe approach: contributions of different features to model representations cannot be directly compared, and feature correlations can affect probing results. We present an Encoding Probe that reverses this direction and reconstructs internal representations of models using interpretable features. We evaluate this method on text and speech transformer models, using feature sets spanning acoustics, phonetics, syntax, lexicon, and speaker identity. Our results suggest that speaker-related effects vary strongly across different training objectives and datasets, while syntactic and lexical features contribute independently to reconstruction. These results show that the Encoding Probe provides a complementary perspective on interpreting model representations beyond decodability. |
| title | Beyond Decodability: Reconstructing Language Model Representations with an Encoding Probe |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2605.00607 |