STLM Engineering Report: Dropout
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929492250853376 |
|---|---|
| author | Hillier, Dylan Guertler, Leon Cheng, Bobby Tan, Cheston |
| author_facet | Hillier, Dylan Guertler, Leon Cheng, Bobby Tan, Cheston |
| contents | In this work we explore the relevance of dropout for modern language models, particularly in the context of models on the scale of <100M parameters. We explore it's relevance firstly in the regime of improving the sample efficiency of models given small, high quality datasets, and secondly in the regime of improving the quality of its fit on larger datasets where models may underfit. We find that concordant with conventional wisdom, dropout remains effective in the overfitting scenario, and that furthermore it may have some relevance for improving the fit of models even in the case of excess data, as suggested by previous research. In the process we find that the existing explanation for the mechanism behind this performance gain is not applicable in the case of language modelling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_05423 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | STLM Engineering Report: Dropout Hillier, Dylan Guertler, Leon Cheng, Bobby Tan, Cheston Computation and Language I.2.7 In this work we explore the relevance of dropout for modern language models, particularly in the context of models on the scale of <100M parameters. We explore it's relevance firstly in the regime of improving the sample efficiency of models given small, high quality datasets, and secondly in the regime of improving the quality of its fit on larger datasets where models may underfit. We find that concordant with conventional wisdom, dropout remains effective in the overfitting scenario, and that furthermore it may have some relevance for improving the fit of models even in the case of excess data, as suggested by previous research. In the process we find that the existing explanation for the mechanism behind this performance gain is not applicable in the case of language modelling. |
| title | STLM Engineering Report: Dropout |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2409.05423 |