Why Is Attention Sparse In Particle Transformer?
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914174525767680 |
|---|---|
| author | Legge, Timothy Wang, Aaron Ortiz, Jacob Limouzi, Victor Zhao, Zihan Gandrakota, Abhijith Khoda, Elham E. Ngadiuba, Jennifer Duarte, Javier Cavanaugh, Richard |
| author_facet | Legge, Timothy Wang, Aaron Ortiz, Jacob Limouzi, Victor Zhao, Zihan Gandrakota, Abhijith Khoda, Elham E. Ngadiuba, Jennifer Duarte, Javier Cavanaugh, Richard |
| contents | Transformer-based models have achieved state-of-the-art performance in jet tagging at the CERN Large Hadron Collider (LHC), with the Particle Transformer (ParT) representing a leading example of such models. A striking feature of ParT is its sparse, nearly binary, attention structure, raising questions about the origin of this behavior and whether it encodes physically meaningful correlations. In this work, we investigate the source of ParT's sparse attention by comparing models trained on multiple benchmark datasets and examine the relative contributions of the attention term and the physics-inspired interaction matrix before softmax. We find that binary sparsity arises primarily from the attention mechanism itself, with the interaction matrix playing a secondary role. Moreove, we show that ParT is able to identify key jet substructure elements, such as leptons in semileptonic top decays, even without explicit particle identification inputs. These results provide new insight into the interpretability of transformer-based jet taggers and clarify the conditions under which sparse attention patterns emerge in ParT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00210 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Why Is Attention Sparse In Particle Transformer? Legge, Timothy Wang, Aaron Ortiz, Jacob Limouzi, Victor Zhao, Zihan Gandrakota, Abhijith Khoda, Elham E. Ngadiuba, Jennifer Duarte, Javier Cavanaugh, Richard High Energy Physics - Phenomenology High Energy Physics - Experiment Data Analysis, Statistics and Probability Transformer-based models have achieved state-of-the-art performance in jet tagging at the CERN Large Hadron Collider (LHC), with the Particle Transformer (ParT) representing a leading example of such models. A striking feature of ParT is its sparse, nearly binary, attention structure, raising questions about the origin of this behavior and whether it encodes physically meaningful correlations. In this work, we investigate the source of ParT's sparse attention by comparing models trained on multiple benchmark datasets and examine the relative contributions of the attention term and the physics-inspired interaction matrix before softmax. We find that binary sparsity arises primarily from the attention mechanism itself, with the interaction matrix playing a secondary role. Moreove, we show that ParT is able to identify key jet substructure elements, such as leptons in semileptonic top decays, even without explicit particle identification inputs. These results provide new insight into the interpretability of transformer-based jet taggers and clarify the conditions under which sparse attention patterns emerge in ParT. |
| title | Why Is Attention Sparse In Particle Transformer? |
| topic | High Energy Physics - Phenomenology High Energy Physics - Experiment Data Analysis, Statistics and Probability |
| url | https://arxiv.org/abs/2512.00210 |