Semantic Convergence: Investigating Shared Representations Across Scaled LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918108032139264 |
|---|---|
| author | Son, Daniel Rathore, Sanjana Rufail, Andrew Simon, Adrian Zhang, Daniel Dave, Soham Blondin, Cole Zhu, Kevin O'Brien, Sean |
| author_facet | Son, Daniel Rathore, Sanjana Rufail, Andrew Simon, Adrian Zhang, Daniel Dave, Soham Blondin, Cole Zhu, Kevin O'Brien, Sean |
| contents | We investigate feature universality in Gemma-2 language models (Gemma-2-2B and Gemma-2-9B), asking whether models with a four-fold difference in scale still converge on comparable internal concepts. Using the Sparse Autoencoder (SAE) dictionary-learning pipeline, we utilize SAEs on each model's residual-stream activations, align the resulting monosemantic features via activation correlation, and compare the matched feature spaces with SVCCA and RSA. Middle layers yield the strongest overlap, while early and late layers show far less similarity. Preliminary experiments extend the analysis from single tokens to multi-token subspaces, showing that semantically similar subspaces interact similarly with language models. These results strengthen the case that large language models carve the world into broadly similar, interpretable features despite size differences, reinforcing universality as a foundation for cross-model interpretability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_22918 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Semantic Convergence: Investigating Shared Representations Across Scaled LLMs Son, Daniel Rathore, Sanjana Rufail, Andrew Simon, Adrian Zhang, Daniel Dave, Soham Blondin, Cole Zhu, Kevin O'Brien, Sean Computation and Language Machine Learning 68T50 I.2.6; I.2.7 We investigate feature universality in Gemma-2 language models (Gemma-2-2B and Gemma-2-9B), asking whether models with a four-fold difference in scale still converge on comparable internal concepts. Using the Sparse Autoencoder (SAE) dictionary-learning pipeline, we utilize SAEs on each model's residual-stream activations, align the resulting monosemantic features via activation correlation, and compare the matched feature spaces with SVCCA and RSA. Middle layers yield the strongest overlap, while early and late layers show far less similarity. Preliminary experiments extend the analysis from single tokens to multi-token subspaces, showing that semantically similar subspaces interact similarly with language models. These results strengthen the case that large language models carve the world into broadly similar, interpretable features despite size differences, reinforcing universality as a foundation for cross-model interpretability. |
| title | Semantic Convergence: Investigating Shared Representations Across Scaled LLMs |
| topic | Computation and Language Machine Learning 68T50 I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2507.22918 |