Topology-Aware Attention: Injecting Structural Priors into Transformer Attention Mechanisms
Fuente:
Zenodo
Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Recurso digital |
| Publicado: |
Zenodo
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866901540855349248 |
|---|---|
| author | Wang, Mingpei |
| author_facet | Wang, Mingpei |
| contents | <p>We propose Topology-Aware Attention, a novel paradigm that injects topological structural priors into Transformer attention mechanisms. While standard Transformers compute attention weights solely based on semantic similarity between tokens, our approach incorporates topological centrality measures to identify and emphasize structurally important tokens. </p> <p>Key findings:<br>- Topological features provide 99.6% independent information from semantic embeddings<br>- Topology-enhanced attention significantly increases the correlation between structural importance and attention received (from r=0.41 to r=0.81)<br>- We propose four injection schemes: Attention Bias, Gated Fusion, Centrality Weighting, and Dual-Stream Attention</p> <p>This work was conducted independently by an 18-year-old researcher, representing the first application of topological priors to Transformer attention mechanisms.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18060955 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Topology-Aware Attention: Injecting Structural Priors into Transformer Attention Mechanisms Wang, Mingpei Transformer Attention Mechanism Topology Persistent Homology Deep Learning Natural Language Processing Topological Data Analysis Machine Learning <p>We propose Topology-Aware Attention, a novel paradigm that injects topological structural priors into Transformer attention mechanisms. While standard Transformers compute attention weights solely based on semantic similarity between tokens, our approach incorporates topological centrality measures to identify and emphasize structurally important tokens. </p> <p>Key findings:<br>- Topological features provide 99.6% independent information from semantic embeddings<br>- Topology-enhanced attention significantly increases the correlation between structural importance and attention received (from r=0.41 to r=0.81)<br>- We propose four injection schemes: Attention Bias, Gated Fusion, Centrality Weighting, and Dual-Stream Attention</p> <p>This work was conducted independently by an 18-year-old researcher, representing the first application of topological priors to Transformer attention mechanisms.</p> |
| title | Topology-Aware Attention: Injecting Structural Priors into Transformer Attention Mechanisms |
| topic | Transformer Attention Mechanism Topology Persistent Homology Deep Learning Natural Language Processing Topological Data Analysis Machine Learning |
| url | https://doi.org/10.5281/zenodo.18060955 |