When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929320983789568 |
|---|---|
| author | Choi, Stephen Gazeley, William |
| author_facet | Choi, Stephen Gazeley, William |
| contents | This paper presents the LLM-ADE framework, a novel methodology for continued pre-training of large language models (LLMs) that addresses the challenges of catastrophic forgetting and double descent. LLM-ADE employs dynamic architectural adjustments, including selective block freezing and expansion, tailored to specific datasets. This strategy enhances model adaptability to new data while preserving previously acquired knowledge. We demonstrate LLM-ADE's effectiveness on the TinyLlama model across various general knowledge benchmarks, showing significant performance improvements without the drawbacks of traditional continuous training methods. This approach promises a more versatile and robust way to keep LLMs current and efficient in real-world applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_13028 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering Choi, Stephen Gazeley, William Computational Engineering, Finance, and Science Artificial Intelligence This paper presents the LLM-ADE framework, a novel methodology for continued pre-training of large language models (LLMs) that addresses the challenges of catastrophic forgetting and double descent. LLM-ADE employs dynamic architectural adjustments, including selective block freezing and expansion, tailored to specific datasets. This strategy enhances model adaptability to new data while preserving previously acquired knowledge. We demonstrate LLM-ADE's effectiveness on the TinyLlama model across various general knowledge benchmarks, showing significant performance improvements without the drawbacks of traditional continuous training methods. This approach promises a more versatile and robust way to keep LLMs current and efficient in real-world applications. |
| title | When Life gives you LLMs, make LLM-ADE: Large Language Models with Adaptive Data Engineering |
| topic | Computational Engineering, Finance, and Science Artificial Intelligence |
| url | https://arxiv.org/abs/2404.13028 |