Quantifying the Importance of Data Alignment in Downstream Model Performance
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915369862561792 |
|---|---|
| author | Chawla, Krrish Sahai, Aryan DePavia, Mario Sundar, Sudharsan Miranda, Brando Obbad, Elyas Koyejo, Sanmi |
| author_facet | Chawla, Krrish Sahai, Aryan DePavia, Mario Sundar, Sudharsan Miranda, Brando Obbad, Elyas Koyejo, Sanmi |
| contents | Contrary to the conventional emphasis on dataset size, we explore the role of data alignment -- an often overlooked aspect of data quality -- in training capable Large Language Models (LLMs). To do so, we use the Task2Vec-based alignment coefficient, a quantitative measure of the similarity between two datasets, to quantify the impact of alignment between training data and evaluation data on downstream performance. In particular, we conduct controlled \textit{interventional} experiments for two settings: 1. the impact of increased alignment coefficients between various pre-training (pt) against evaluation datasets, and 2. the impact of increased alignment coefficients between domain specific fine-tuning (ft) against domain specific evaluation. The domain specific task we explore is Autoformalization -- the machine translation task between natural language and code for formal verification. In both settings, we find a strong, predictable negative correlation between the alignment coefficient of a model's training and evaluation data and the model's loss/perplexity on the respective downstream task. These findings suggest a re-evaluation of LLM training approaches, demonstrating the relevance of data alignment compared to data quantity, especially in specialized downstream tasks such as Autoformalization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08496 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Quantifying the Importance of Data Alignment in Downstream Model Performance Chawla, Krrish Sahai, Aryan DePavia, Mario Sundar, Sudharsan Miranda, Brando Obbad, Elyas Koyejo, Sanmi Computation and Language Artificial Intelligence Machine Learning Programming Languages Contrary to the conventional emphasis on dataset size, we explore the role of data alignment -- an often overlooked aspect of data quality -- in training capable Large Language Models (LLMs). To do so, we use the Task2Vec-based alignment coefficient, a quantitative measure of the similarity between two datasets, to quantify the impact of alignment between training data and evaluation data on downstream performance. In particular, we conduct controlled \textit{interventional} experiments for two settings: 1. the impact of increased alignment coefficients between various pre-training (pt) against evaluation datasets, and 2. the impact of increased alignment coefficients between domain specific fine-tuning (ft) against domain specific evaluation. The domain specific task we explore is Autoformalization -- the machine translation task between natural language and code for formal verification. In both settings, we find a strong, predictable negative correlation between the alignment coefficient of a model's training and evaluation data and the model's loss/perplexity on the respective downstream task. These findings suggest a re-evaluation of LLM training approaches, demonstrating the relevance of data alignment compared to data quantity, especially in specialized downstream tasks such as Autoformalization. |
| title | Quantifying the Importance of Data Alignment in Downstream Model Performance |
| topic | Computation and Language Artificial Intelligence Machine Learning Programming Languages |
| url | https://arxiv.org/abs/2501.08496 |