Accelerate Scaling of LLM Finetuning via Quantifying the Coverage and Depth of Instruction Set
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908616536096768 |
|---|---|
| author | Wu, Chengwei Du, Li Zhao, Hanyu Ju, Yiming Wang, Jiapu Chen, Tianyu Zhou, Haoyi |
| author_facet | Wu, Chengwei Du, Li Zhao, Hanyu Ju, Yiming Wang, Jiapu Chen, Tianyu Zhou, Haoyi |
| contents | Scaling the amount of data used for supervied fine-tuning(SFT) does not guarantee the proportional gains in model performance, highlighting a critical need to understand what makes training samples effective. This work identifies two fundamental dataset properties that govern SFT scalability: \textbf{semantic coverage}, or the breadth of task domains, and \textbf{information depth}, or the richness of individual examples. We demonstrate that simple proxies for these properties explain the majority of validation loss variance in our experiments. In this work, we further propose the \textbf{Information Landscape Approximation (ILA)}, a model-agnostic data selection framework that jointly optimizes for these two factors. ILA constructs compact subsets that approximate the informational value of large datasets. Empirical results show that models tuned on ILA-selected data achieve faster and more sustained performance improvements across diverse tasks and model sizes compared to existing methods, a phenomenon we term \textbf{accelerated scaling}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_06463 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Accelerate Scaling of LLM Finetuning via Quantifying the Coverage and Depth of Instruction Set Wu, Chengwei Du, Li Zhao, Hanyu Ju, Yiming Wang, Jiapu Chen, Tianyu Zhou, Haoyi Artificial Intelligence Scaling the amount of data used for supervied fine-tuning(SFT) does not guarantee the proportional gains in model performance, highlighting a critical need to understand what makes training samples effective. This work identifies two fundamental dataset properties that govern SFT scalability: \textbf{semantic coverage}, or the breadth of task domains, and \textbf{information depth}, or the richness of individual examples. We demonstrate that simple proxies for these properties explain the majority of validation loss variance in our experiments. In this work, we further propose the \textbf{Information Landscape Approximation (ILA)}, a model-agnostic data selection framework that jointly optimizes for these two factors. ILA constructs compact subsets that approximate the informational value of large datasets. Empirical results show that models tuned on ILA-selected data achieve faster and more sustained performance improvements across diverse tasks and model sizes compared to existing methods, a phenomenon we term \textbf{accelerated scaling}. |
| title | Accelerate Scaling of LLM Finetuning via Quantifying the Coverage and Depth of Instruction Set |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.06463 |