Terminating Differentiable Tree Experts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917710332428288 |
|---|---|
| author | Thomm, Jonathan Hersche, Michael Camposampiero, Giacomo Terzić, Aleksandar Schölkopf, Bernhard Rahimi, Abbas |
| author_facet | Thomm, Jonathan Hersche, Michael Camposampiero, Giacomo Terzić, Aleksandar Schölkopf, Bernhard Rahimi, Abbas |
| contents | We advance the recently proposed neuro-symbolic Differentiable Tree Machine, which learns tree operations using a combination of transformers and Tensor Product Representations. We investigate the architecture and propose two key components. We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts. This results in a Differentiable Tree Experts model with a constant number of parameters for any arbitrary number of steps in the computation, compared to the previous method in the Differentiable Tree Machine with a linear growth. Given this flexibility in the number of steps, we additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically. The resulting Terminating Differentiable Tree Experts model sluggishly learns to predict the number of steps without an oracle. It can do so while maintaining the learning capabilities of the model, converging to the optimal amount of steps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_02060 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Terminating Differentiable Tree Experts Thomm, Jonathan Hersche, Michael Camposampiero, Giacomo Terzić, Aleksandar Schölkopf, Bernhard Rahimi, Abbas Machine Learning Artificial Intelligence Symbolic Computation We advance the recently proposed neuro-symbolic Differentiable Tree Machine, which learns tree operations using a combination of transformers and Tensor Product Representations. We investigate the architecture and propose two key components. We first remove a series of different transformer layers that are used in every step by introducing a mixture of experts. This results in a Differentiable Tree Experts model with a constant number of parameters for any arbitrary number of steps in the computation, compared to the previous method in the Differentiable Tree Machine with a linear growth. Given this flexibility in the number of steps, we additionally propose a new termination algorithm to provide the model the power to choose how many steps to make automatically. The resulting Terminating Differentiable Tree Experts model sluggishly learns to predict the number of steps without an oracle. It can do so while maintaining the learning capabilities of the model, converging to the optimal amount of steps. |
| title | Terminating Differentiable Tree Experts |
| topic | Machine Learning Artificial Intelligence Symbolic Computation |
| url | https://arxiv.org/abs/2407.02060 |