Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.15347 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916870514278400 |
|---|---|
| author | Buonanno, Amedeo Rivetti, Alessandro Palmieri, Francesco A. N. Di Gennaro, Giovanni Romano, Gianmarco |
| author_facet | Buonanno, Amedeo Rivetti, Alessandro Palmieri, Francesco A. N. Di Gennaro, Giovanni Romano, Gianmarco |
| contents | This work explores entropy analysis as a tool for probing information distribution within Transformer-based architectures. By quantifying token-level uncertainty and examining entropy patterns across different stages of processing, we aim to investigate how information is managed and transformed within these models. As a case study, we apply the methodology to a GPT-based large language model, illustrating its potential to reveal insights into model behavior and internal representations. This approach may offer insights into model behavior and contribute to the development of interpretability and evaluation frameworks for transformer-based models |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15347 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Probing Information Distribution in Transformer Architectures through Entropy Analysis Buonanno, Amedeo Rivetti, Alessandro Palmieri, Francesco A. N. Di Gennaro, Giovanni Romano, Gianmarco Computation and Language Machine Learning This work explores entropy analysis as a tool for probing information distribution within Transformer-based architectures. By quantifying token-level uncertainty and examining entropy patterns across different stages of processing, we aim to investigate how information is managed and transformed within these models. As a case study, we apply the methodology to a GPT-based large language model, illustrating its potential to reveal insights into model behavior and internal representations. This approach may offer insights into model behavior and contribute to the development of interpretability and evaluation frameworks for transformer-based models |
| title | Probing Information Distribution in Transformer Architectures through Entropy Analysis |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.15347 |