On Implications of Scaling Laws on Feature Superposition
Fuente:
arXiv
Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913411942580224 |
|---|---|
| author | Katta, Pavan |
| author_facet | Katta, Pavan |
| contents | Using results from scaling laws, this theoretical note argues that the following two statements cannot be simultaneously true: 1. Superposition hypothesis where sparse features are linearly represented across a layer is a complete theory of feature representation. 2. Features are universal, meaning two models trained on the same data and achieving equal performance will learn identical features. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01459 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Implications of Scaling Laws on Feature Superposition Katta, Pavan Machine Learning Artificial Intelligence Using results from scaling laws, this theoretical note argues that the following two statements cannot be simultaneously true: 1. Superposition hypothesis where sparse features are linearly represented across a layer is a complete theory of feature representation. 2. Features are universal, meaning two models trained on the same data and achieving equal performance will learn identical features. |
| title | On Implications of Scaling Laws on Feature Superposition |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2407.01459 |