Disentangling the Roles of Representation and Selection in Data Pruning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909675856855040 |
|---|---|
| author | Du, Yupei Song, Yingjin Wong, Hugh Mee Ignatev, Daniil Gatt, Albert Nguyen, Dong |
| author_facet | Du, Yupei Song, Yingjin Wong, Hugh Mee Ignatev, Daniil Gatt, Albert Nguyen, Dong |
| contents | Data pruning, selecting small but impactful subsets, offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design choices, which have not been systematically studied. This limits future developments. In this work, we decompose data pruning into two key components: the data representation and the selection algorithm, and we systematically analyze their influence on the selection of instances. Our theoretical and empirical results highlight the crucial role of representations: better representations, e.g., training gradients, generally lead to a better selection of instances, regardless of the chosen selection algorithm. Furthermore, different selection algorithms excel in different settings, and none consistently outperforms the others. Moreover, the selection algorithms do not always align with their intended objectives: for example, algorithms designed for the same objective can select drastically different instances, highlighting the need for careful evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_03648 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Disentangling the Roles of Representation and Selection in Data Pruning Du, Yupei Song, Yingjin Wong, Hugh Mee Ignatev, Daniil Gatt, Albert Nguyen, Dong Computation and Language Machine Learning Data pruning, selecting small but impactful subsets, offers a promising way to efficiently scale NLP model training. However, existing methods often involve many different design choices, which have not been systematically studied. This limits future developments. In this work, we decompose data pruning into two key components: the data representation and the selection algorithm, and we systematically analyze their influence on the selection of instances. Our theoretical and empirical results highlight the crucial role of representations: better representations, e.g., training gradients, generally lead to a better selection of instances, regardless of the chosen selection algorithm. Furthermore, different selection algorithms excel in different settings, and none consistently outperforms the others. Moreover, the selection algorithms do not always align with their intended objectives: for example, algorithms designed for the same objective can select drastically different instances, highlighting the need for careful evaluation. |
| title | Disentangling the Roles of Representation and Selection in Data Pruning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2507.03648 |