The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909135720677376 |
|---|---|
| author | Nicolini, Carlo Staiano, Jacopo Lepri, Bruno Marino, Raffaele |
| author_facet | Nicolini, Carlo Staiano, Jacopo Lepri, Bruno Marino, Raffaele |
| contents | A substantial gap persists in understanding the reasons behind the exceptional performance of the Transformer architecture in NLP. A particularly unexplored area involves the mechanistic description of how the distribution of parameters evolves over time during training. In this work we suggest that looking at the time evolution of the statistic distribution of model parameters, and specifically at bifurcation effects, can help understanding the model quality, potentially reducing training costs and evaluation efforts and empirically showing the reasons behind the effectiveness of weights sparsification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_08739 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models Nicolini, Carlo Staiano, Jacopo Lepri, Bruno Marino, Raffaele Computation and Language Disordered Systems and Neural Networks Statistical Mechanics Artificial Intelligence A substantial gap persists in understanding the reasons behind the exceptional performance of the Transformer architecture in NLP. A particularly unexplored area involves the mechanistic description of how the distribution of parameters evolves over time during training. In this work we suggest that looking at the time evolution of the statistic distribution of model parameters, and specifically at bifurcation effects, can help understanding the model quality, potentially reducing training costs and evaluation efforts and empirically showing the reasons behind the effectiveness of weights sparsification. |
| title | The Garden of Forking Paths: Observing Dynamic Parameters Distribution in Large Language Models |
| topic | Computation and Language Disordered Systems and Neural Networks Statistical Mechanics Artificial Intelligence |
| url | https://arxiv.org/abs/2403.08739 |