Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions
Fuente:
arXiv
Guardado en:
| Autor principal: | |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929352877277184 |
|---|---|
| author | Cai, Yongqiang |
| author_facet | Cai, Yongqiang |
| contents | In recent years, deep learning-based sequence modelings, such as language models, have received much attention and success, which pushes researchers to explore the possibility of transforming non-sequential problems into a sequential form. Following this thought, deep neural networks can be represented as composite functions of a sequence of mappings, linear or nonlinear, where each composition can be viewed as a \emph{word}. However, the weights of linear mappings are undetermined and hence require an infinite number of words. In this article, we investigate the finite case and constructively prove the existence of a finite \emph{vocabulary} $V=\{ϕ_i: \mathbb{R}^d \to \mathbb{R}^d | i=1,...,n\}$ with $n=O(d^2)$ for the universal approximation. That is, for any continuous mapping $f: \mathbb{R}^d \to \mathbb{R}^d$, compact domain $Ω$ and $\varepsilon>0$, there is a sequence of mappings $ϕ_{i_1}, ..., ϕ_{i_m} \in V, m \in \mathbb{Z}_+$, such that the composition $ϕ_{i_m} \circ ... \circ ϕ_{i_1} $ approximates $f$ on $Ω$ with an error less than $\varepsilon$. Our results demonstrate an unusual approximation power of mapping compositions and motivate a novel compositional model for regular languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_12205 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions Cai, Yongqiang Machine Learning Numerical Analysis Dynamical Systems In recent years, deep learning-based sequence modelings, such as language models, have received much attention and success, which pushes researchers to explore the possibility of transforming non-sequential problems into a sequential form. Following this thought, deep neural networks can be represented as composite functions of a sequence of mappings, linear or nonlinear, where each composition can be viewed as a \emph{word}. However, the weights of linear mappings are undetermined and hence require an infinite number of words. In this article, we investigate the finite case and constructively prove the existence of a finite \emph{vocabulary} $V=\{ϕ_i: \mathbb{R}^d \to \mathbb{R}^d | i=1,...,n\}$ with $n=O(d^2)$ for the universal approximation. That is, for any continuous mapping $f: \mathbb{R}^d \to \mathbb{R}^d$, compact domain $Ω$ and $\varepsilon>0$, there is a sequence of mappings $ϕ_{i_1}, ..., ϕ_{i_m} \in V, m \in \mathbb{Z}_+$, such that the composition $ϕ_{i_m} \circ ... \circ ϕ_{i_1} $ approximates $f$ on $Ω$ with an error less than $\varepsilon$. Our results demonstrate an unusual approximation power of mapping compositions and motivate a novel compositional model for regular languages. |
| title | Vocabulary for Universal Approximation: A Linguistic Perspective of Mapping Compositions |
| topic | Machine Learning Numerical Analysis Dynamical Systems |
| url | https://arxiv.org/abs/2305.12205 |