Beyond Self Attention: A Subquadratic Fourier Wavelet Transformer with Multi Modal Fusion
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2021
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915253509423104 |
|---|---|
| author | Kiruluta, Andrew Lemos, Andreas Lundy, Eric |
| author_facet | Kiruluta, Andrew Lemos, Andreas Lundy, Eric |
| contents | We revisit the use of spectral techniques to replaces the attention mechanism in Transformers through Fourier Transform based token mixing, and present a comprehensive and novel reformulation of this technique in next generation transformer models. We provide expanded literature context, detailed mathematical formulations of Fourier mixing and causal masking, and introduce a novel MultiDomain Fourier Wavelet Attention(MDFWA) that integrates frequency and time localized transforms to capture both global and local dependencies efficiently. We derive the complexity bounds, gradient formulas, and show that MDFWA achieves sub quadratic time and memory cost while improving expressive power. We validate our design on an abstractive summarization task using PubMed dataset, by enhancing the proposed approach with learned frequency bases, adaptive scale selection, and multi-modal extensions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2111_15473 |
| institution | arXiv |
| publishDate | 2021 |
| record_format | arxiv |
| spellingShingle | Beyond Self Attention: A Subquadratic Fourier Wavelet Transformer with Multi Modal Fusion Kiruluta, Andrew Lemos, Andreas Lundy, Eric Computation and Language Machine Learning We revisit the use of spectral techniques to replaces the attention mechanism in Transformers through Fourier Transform based token mixing, and present a comprehensive and novel reformulation of this technique in next generation transformer models. We provide expanded literature context, detailed mathematical formulations of Fourier mixing and causal masking, and introduce a novel MultiDomain Fourier Wavelet Attention(MDFWA) that integrates frequency and time localized transforms to capture both global and local dependencies efficiently. We derive the complexity bounds, gradient formulas, and show that MDFWA achieves sub quadratic time and memory cost while improving expressive power. We validate our design on an abstractive summarization task using PubMed dataset, by enhancing the proposed approach with learned frequency bases, adaptive scale selection, and multi-modal extensions. |
| title | Beyond Self Attention: A Subquadratic Fourier Wavelet Transformer with Multi Modal Fusion |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2111.15473 |