Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Size
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909496657313792 |
|---|---|
| author | Takeshita, Naoki Imaizumi, Masaaki |
| author_facet | Takeshita, Naoki Imaizumi, Masaaki |
| contents | Transformers are a type of neural network that have demonstrated remarkable performance across various domains, particularly in natural language processing tasks. Motivated by this success, research on the theoretical understanding of transformers has garnered significant attention. A notable example is the mathematical analysis of their approximation power, which validates the empirical expressive capability of transformers. In this study, we investigate the ability of transformers to approximate column-symmetric polynomials, an extension of symmetric polynomials that take matrices as input. Consequently, we establish an explicit relationship between the size of the transformer network and its approximation capability, leveraging the parameter efficiency of transformers and their compatibility with symmetry by focusing on the algebraic properties of symmetric polynomials. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11467 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Size Takeshita, Naoki Imaizumi, Masaaki Machine Learning Functional Analysis Transformers are a type of neural network that have demonstrated remarkable performance across various domains, particularly in natural language processing tasks. Motivated by this success, research on the theoretical understanding of transformers has garnered significant attention. A notable example is the mathematical analysis of their approximation power, which validates the empirical expressive capability of transformers. In this study, we investigate the ability of transformers to approximate column-symmetric polynomials, an extension of symmetric polynomials that take matrices as input. Consequently, we establish an explicit relationship between the size of the transformer network and its approximation capability, leveraging the parameter efficiency of transformers and their compatibility with symmetry by focusing on the algebraic properties of symmetric polynomials. |
| title | Approximation of Permutation Invariant Polynomials by Transformers: Efficient Construction in Column-Size |
| topic | Machine Learning Functional Analysis |
| url | https://arxiv.org/abs/2502.11467 |