Dynamic layer selection in decoder-only transformers
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916455838121984 |
|---|---|
| author | Glavas, Theodore Chataoui, Joud Regol, Florence Jabbour, Wassim Valkanas, Antonios Oreshkin, Boris N. Coates, Mark |
| author_facet | Glavas, Theodore Chataoui, Joud Regol, Florence Jabbour, Wassim Valkanas, Antonios Oreshkin, Boris N. Coates, Mark |
| contents | The vast size of Large Language Models (LLMs) has prompted a search to optimize inference. One effective approach is dynamic inference, which adapts the architecture to the sample-at-hand to reduce the overall computational cost. We empirically examine two common dynamic inference methods for natural language generation (NLG): layer skipping and early exiting. We find that a pre-trained decoder-only model is significantly more robust to layer removal via layer skipping, as opposed to early exit. We demonstrate the difficulty of using hidden state information to adapt computation on a per-token basis for layer skipping. Finally, we show that dynamic computation allocation on a per-sequence basis holds promise for significant efficiency gains by constructing an oracle controller. Remarkably, we find that there exists an allocation which achieves equal performance to the full model using only 23.3% of its layers on average. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_20022 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamic layer selection in decoder-only transformers Glavas, Theodore Chataoui, Joud Regol, Florence Jabbour, Wassim Valkanas, Antonios Oreshkin, Boris N. Coates, Mark Computation and Language Machine Learning The vast size of Large Language Models (LLMs) has prompted a search to optimize inference. One effective approach is dynamic inference, which adapts the architecture to the sample-at-hand to reduce the overall computational cost. We empirically examine two common dynamic inference methods for natural language generation (NLG): layer skipping and early exiting. We find that a pre-trained decoder-only model is significantly more robust to layer removal via layer skipping, as opposed to early exit. We demonstrate the difficulty of using hidden state information to adapt computation on a per-token basis for layer skipping. Finally, we show that dynamic computation allocation on a per-sequence basis holds promise for significant efficiency gains by constructing an oracle controller. Remarkably, we find that there exists an allocation which achieves equal performance to the full model using only 23.3% of its layers on average. |
| title | Dynamic layer selection in decoder-only transformers |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.20022 |