Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917935787802624 |
|---|---|
| author | Song, Yuda Zhang, Hanlin Eisenach, Carson Kakade, Sham Foster, Dean Ghai, Udaya |
| author_facet | Song, Yuda Zhang, Hanlin Eisenach, Carson Kakade, Sham Foster, Dean Ghai, Udaya |
| contents | Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the generation-verification gap. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02674 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models Song, Yuda Zhang, Hanlin Eisenach, Carson Kakade, Sham Foster, Dean Ghai, Udaya Computation and Language Machine Learning Self-improvement is a mechanism in Large Language Model (LLM) pre-training, post-training and test-time inference. We explore a framework where the model verifies its own outputs, filters or reweights data based on this verification, and distills the filtered data. Despite several empirical successes, a fundamental understanding is still lacking. In this work, we initiate a comprehensive, modular and controlled study on LLM self-improvement. We provide a mathematical formulation for self-improvement, which is largely governed by a quantity which we formalize as the generation-verification gap. Through experiments with various model families and tasks, we discover a scaling phenomenon of self-improvement -- a variant of the generation-verification gap scales monotonically with the model pre-training flops. We also examine when self-improvement is possible, an iterative self-improvement procedure, and ways to improve its performance. Our findings not only advance understanding of LLM self-improvement with practical implications, but also open numerous avenues for future research into its capabilities and boundaries. |
| title | Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2412.02674 |