Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916477736583168 |
|---|---|
| author | Shi, Jingze Wu, Bingheng He, Lu Jiang, Luchang |
| author_facet | Shi, Jingze Wu, Bingheng He, Lu Jiang, Luchang |
| contents | We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadratic causal self-attention and state space dual algorithms. We propose inner function attention with dynamic mask, which can improve the expressiveness of the attention algorithm and avoid the sequence noise significantly affecting the accuracy of the attention score. We also design cross domain mixture of experts, which can improve the granularity of the sparse activation feedforward network while maintaining the efficiency of parameter utilization and retrieval. The combination of these methods constitutes our foundation model architecture: Wonderful Matrices. We conduct experiments on the language modeling task and find that Wonderful Matrices are more efficient and effective in handling complex language tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_16958 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks Shi, Jingze Wu, Bingheng He, Lu Jiang, Luchang Machine Learning Artificial Intelligence We prove the availability of inner product form position encoding in the state space dual algorithm and study the effectiveness of different position embeddings in the hybrid quadratic causal self-attention and state space dual algorithms. We propose inner function attention with dynamic mask, which can improve the expressiveness of the attention algorithm and avoid the sequence noise significantly affecting the accuracy of the attention score. We also design cross domain mixture of experts, which can improve the granularity of the sparse activation feedforward network while maintaining the efficiency of parameter utilization and retrieval. The combination of these methods constitutes our foundation model architecture: Wonderful Matrices. We conduct experiments on the language modeling task and find that Wonderful Matrices are more efficient and effective in handling complex language tasks. |
| title | Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2407.16958 |