Preserving Bilinear Weight Spectra with a Signed and Shrunk Quadratic Activation Function
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912566269181952 |
|---|---|
| author | Abohwo, Jason Mosen, Thomas |
| author_facet | Abohwo, Jason Mosen, Thomas |
| contents | Understanding the inner workings of machine learning models is critical for ensuring their reliability and robustness. Whilst many techniques in mechanistic interpretability focus on activation driven analyses, being able to derive meaningful features directly from the weights of a neural network would provide greater guarantees and more computational efficiency. Existing techniques for analyzing model features through weights suffer from drawbacks such as reduced performance and data inefficiency. In this paper, we introduce Signed Quadratic Shrink (SQS), an activation function designed to allow Gated Linear Units (GLUs) to learn interpretable features without these drawbacks. Our experimental results show that SQS achieves performance competitive with state-of-the-art activation functions whilst enabling weight-based interpretability |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_01874 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Preserving Bilinear Weight Spectra with a Signed and Shrunk Quadratic Activation Function Abohwo, Jason Mosen, Thomas Machine Learning Artificial Intelligence Understanding the inner workings of machine learning models is critical for ensuring their reliability and robustness. Whilst many techniques in mechanistic interpretability focus on activation driven analyses, being able to derive meaningful features directly from the weights of a neural network would provide greater guarantees and more computational efficiency. Existing techniques for analyzing model features through weights suffer from drawbacks such as reduced performance and data inefficiency. In this paper, we introduce Signed Quadratic Shrink (SQS), an activation function designed to allow Gated Linear Units (GLUs) to learn interpretable features without these drawbacks. Our experimental results show that SQS achieves performance competitive with state-of-the-art activation functions whilst enabling weight-based interpretability |
| title | Preserving Bilinear Weight Spectra with a Signed and Shrunk Quadratic Activation Function |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2509.01874 |