Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.06985 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909641304178688 |
|---|---|
| author | Zhao, Wenhao Xu, Qiushui Xu, Linjie Song, Lei Wang, Jinyu Zhou, Chunlai Bian, Jiang |
| author_facet | Zhao, Wenhao Xu, Qiushui Xu, Linjie Song, Lei Wang, Jinyu Zhou, Chunlai Bian, Jiang |
| contents | Recently, incorporating knowledge from pretrained language models (PLMs) into decision transformers (DTs) has generated significant attention in offline reinforcement learning (RL). These PLMs perform well in RL tasks, raising an intriguing question: what kind of knowledge from PLMs has been transferred to RL to achieve such good results? This work first dives into this problem by analyzing each head quantitatively and points out Markov head, a crucial component that exists in the attention heads of PLMs. It leads to extreme attention on the last-input token and performs well only in short-term environments. Furthermore, we prove that this extreme attention cannot be changed by re-training embedding layer or fine-tuning. Inspired by our analysis, we propose a general method GPT2-DTMA, which equips a pretrained DT with Mixture of Attention (MoA), to accommodate diverse attention requirements during fine-tuning. Extensive experiments corroborate our theorems and demonstrate the effectiveness of GPT2-DTMA: it achieves comparable performance in short-term environments while significantly narrowing the performance gap in long-term environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_06985 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learning Zhao, Wenhao Xu, Qiushui Xu, Linjie Song, Lei Wang, Jinyu Zhou, Chunlai Bian, Jiang Machine Learning Recently, incorporating knowledge from pretrained language models (PLMs) into decision transformers (DTs) has generated significant attention in offline reinforcement learning (RL). These PLMs perform well in RL tasks, raising an intriguing question: what kind of knowledge from PLMs has been transferred to RL to achieve such good results? This work first dives into this problem by analyzing each head quantitatively and points out Markov head, a crucial component that exists in the attention heads of PLMs. It leads to extreme attention on the last-input token and performs well only in short-term environments. Furthermore, we prove that this extreme attention cannot be changed by re-training embedding layer or fine-tuning. Inspired by our analysis, we propose a general method GPT2-DTMA, which equips a pretrained DT with Mixture of Attention (MoA), to accommodate diverse attention requirements during fine-tuning. Extensive experiments corroborate our theorems and demonstrate the effectiveness of GPT2-DTMA: it achieves comparable performance in short-term environments while significantly narrowing the performance gap in long-term environments. |
| title | Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2409.06985 |