Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915174568427520 |
|---|---|
| author | Sokar, Ghada Obando-Ceron, Johan Courville, Aaron Larochelle, Hugo Castro, Pablo Samuel |
| author_facet | Sokar, Ghada Obando-Ceron, Johan Courville, Aaron Larochelle, Hugo Castro, Pablo Samuel |
| contents | The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have recently shown promise in mitigating this issue for online RL, the reasons behind their effectiveness remain largely unknown. In this work we provide an in-depth analysis identifying the key factors driving this performance gain. We discover the surprising result that tokenizing the encoder output, rather than the use of multiple experts, is what is behind the efficacy of SoftMoEs. Indeed, we demonstrate that even with an appropriately scaled single expert, we are able to maintain the performance gains, largely thanks to tokenization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01930 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL Sokar, Ghada Obando-Ceron, Johan Courville, Aaron Larochelle, Hugo Castro, Pablo Samuel Machine Learning Artificial Intelligence The use of deep neural networks in reinforcement learning (RL) often suffers from performance degradation as model size increases. While soft mixtures of experts (SoftMoEs) have recently shown promise in mitigating this issue for online RL, the reasons behind their effectiveness remain largely unknown. In this work we provide an in-depth analysis identifying the key factors driving this performance gain. We discover the surprising result that tokenizing the encoder output, rather than the use of multiple experts, is what is behind the efficacy of SoftMoEs. Indeed, we demonstrate that even with an appropriately scaled single expert, we are able to maintain the performance gains, largely thanks to tokenization. |
| title | Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RL |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2410.01930 |