Generalized Holographic Reduced Representations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910261669003264 |
|---|---|
| author | Yeung, Calvin Zou, Zhuowen Jeong, SungHeon Huang, Wenjun Bastian, Nathaniel D Imani, Mohsen |
| author_facet | Yeung, Calvin Zou, Zhuowen Jeong, SungHeon Huang, Wenjun Bastian, Nathaniel D Imani, Mohsen |
| contents | Hyperdimensional Computing (HDC) is a computationally and data-efficient paradigm that acts as a bridge between connectionist and symbolic approaches to artificial intelligence (AI). However, HDC's simplicity poses challenges for encoding complex compositional structures, especially in its binding operation. To address this, we propose Generalized Holographic Reduced Representations (GHRR), an extension of Fourier Holographic Reduced Representations (FHRR), a specific HDC implementation. GHRR introduces a flexible, non-commutative binding operation, enabling improved encoding of complex data structures while preserving HDC's desirable properties of robustness and transparency. In this work, we introduce the GHRR framework, prove its theoretical properties and its adherence to HDC properties, explore its kernel and binding characteristics, and perform empirical experiments showcasing its flexible non-commutativity, enhanced decoding accuracy for compositional structures. We also demonstrate that binding in GHRR is more expressive than that in other HDC variants; in particular, we show that binding in GHRR can implement a kind of attention mechanism. We verify this by replacing the attention mechanism in a transformer with its GHRR-equivalent and testing it on a language modeling task, showing improved performance compared to a vanilla transformer. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_09689 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generalized Holographic Reduced Representations Yeung, Calvin Zou, Zhuowen Jeong, SungHeon Huang, Wenjun Bastian, Nathaniel D Imani, Mohsen Machine Learning Artificial Intelligence Symbolic Computation Hyperdimensional Computing (HDC) is a computationally and data-efficient paradigm that acts as a bridge between connectionist and symbolic approaches to artificial intelligence (AI). However, HDC's simplicity poses challenges for encoding complex compositional structures, especially in its binding operation. To address this, we propose Generalized Holographic Reduced Representations (GHRR), an extension of Fourier Holographic Reduced Representations (FHRR), a specific HDC implementation. GHRR introduces a flexible, non-commutative binding operation, enabling improved encoding of complex data structures while preserving HDC's desirable properties of robustness and transparency. In this work, we introduce the GHRR framework, prove its theoretical properties and its adherence to HDC properties, explore its kernel and binding characteristics, and perform empirical experiments showcasing its flexible non-commutativity, enhanced decoding accuracy for compositional structures. We also demonstrate that binding in GHRR is more expressive than that in other HDC variants; in particular, we show that binding in GHRR can implement a kind of attention mechanism. We verify this by replacing the attention mechanism in a transformer with its GHRR-equivalent and testing it on a language modeling task, showing improved performance compared to a vanilla transformer. |
| title | Generalized Holographic Reduced Representations |
| topic | Machine Learning Artificial Intelligence Symbolic Computation |
| url | https://arxiv.org/abs/2405.09689 |