Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913245367894016 |
|---|---|
| author | Axiotis, Kyriakos Cohen-Addad, Vincent Henzinger, Monika Jerome, Sammy Mirrokni, Vahab Saulpic, David Woodruff, David Wunder, Michael |
| author_facet | Axiotis, Kyriakos Cohen-Addad, Vincent Henzinger, Monika Jerome, Sammy Mirrokni, Vahab Saulpic, David Woodruff, David Wunder, Michael |
| contents | We study the data selection problem, whose aim is to select a small representative subset of data that can be used to efficiently train a machine learning model. We present a new data selection approach based on $k$-means clustering and sensitivity sampling. Assuming access to an embedding representation of the data with respect to which the model loss is Hölder continuous, our approach provably allows selecting a set of ``typical'' $k + 1/\varepsilon^2$ elements whose average loss corresponds to the average loss of the whole dataset, up to a multiplicative $(1\pm\varepsilon)$ factor and an additive $\varepsilon λΦ_k$, where $Φ_k$ represents the $k$-means cost for the input embeddings and $λ$ is the Hölder constant.
We furthermore demonstrate the performance and scalability of our approach on fine-tuning foundation models and show that it outperforms state-of-the-art methods. We also show how it can be applied on linear regression, leading to a new sampling strategy that surprisingly matches the performances of leverage score sampling, while being conceptually simpler and more scalable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_17327 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond Axiotis, Kyriakos Cohen-Addad, Vincent Henzinger, Monika Jerome, Sammy Mirrokni, Vahab Saulpic, David Woodruff, David Wunder, Michael Machine Learning Data Structures and Algorithms We study the data selection problem, whose aim is to select a small representative subset of data that can be used to efficiently train a machine learning model. We present a new data selection approach based on $k$-means clustering and sensitivity sampling. Assuming access to an embedding representation of the data with respect to which the model loss is Hölder continuous, our approach provably allows selecting a set of ``typical'' $k + 1/\varepsilon^2$ elements whose average loss corresponds to the average loss of the whole dataset, up to a multiplicative $(1\pm\varepsilon)$ factor and an additive $\varepsilon λΦ_k$, where $Φ_k$ represents the $k$-means cost for the input embeddings and $λ$ is the Hölder constant. We furthermore demonstrate the performance and scalability of our approach on fine-tuning foundation models and show that it outperforms state-of-the-art methods. We also show how it can be applied on linear regression, leading to a new sampling strategy that surprisingly matches the performances of leverage score sampling, while being conceptually simpler and more scalable. |
| title | Data-Efficient Learning via Clustering-Based Sensitivity Sampling: Foundation Models and Beyond |
| topic | Machine Learning Data Structures and Algorithms |
| url | https://arxiv.org/abs/2402.17327 |