Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.01706 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911056889118720 |
|---|---|
| author | Figueroa, Alexei Westerhoff, Justus Atefi, Golzar Fast, Dennis Winter, Benjamin Gers, Felix Alexander Löser, Alexander Nejdl, Wolfgang |
| author_facet | Figueroa, Alexei Westerhoff, Justus Atefi, Golzar Fast, Dennis Winter, Benjamin Gers, Felix Alexander Löser, Alexander Nejdl, Wolfgang |
| contents | Biologically inspired neural networks offer alternative avenues to model data distributions. FlyVec is a recent example that draws inspiration from the fruit fly's olfactory circuit to tackle the task of learning word embeddings. Surprisingly, this model performs competitively even against deep learning approaches specifically designed to encode text, and it does so with the highest degree of computational efficiency. We pose the question of whether this performance can be improved further. For this, we introduce Comply. By incorporating positional information through complex weights, we enable a single-layer neural network to learn sequence representations. Our experiments show that Comply not only supersedes FlyVec but also performs on par with significantly larger state-of-the-art models. We achieve this without additional parameters. Comply yields sparse contextual representations of sentences that can be interpreted explicitly from the neuron weights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01706 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfaction Figueroa, Alexei Westerhoff, Justus Atefi, Golzar Fast, Dennis Winter, Benjamin Gers, Felix Alexander Löser, Alexander Nejdl, Wolfgang Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing Biologically inspired neural networks offer alternative avenues to model data distributions. FlyVec is a recent example that draws inspiration from the fruit fly's olfactory circuit to tackle the task of learning word embeddings. Surprisingly, this model performs competitively even against deep learning approaches specifically designed to encode text, and it does so with the highest degree of computational efficiency. We pose the question of whether this performance can be improved further. For this, we introduce Comply. By incorporating positional information through complex weights, we enable a single-layer neural network to learn sequence representations. Our experiments show that Comply not only supersedes FlyVec but also performs on par with significantly larger state-of-the-art models. We achieve this without additional parameters. Comply yields sparse contextual representations of sentences that can be interpreted explicitly from the neuron weights. |
| title | Comply: Learning Sentences with Complex Weights inspired by Fruit Fly Olfaction |
| topic | Computation and Language Artificial Intelligence Machine Learning Neural and Evolutionary Computing |
| url | https://arxiv.org/abs/2502.01706 |