DataS^3: Dataset Subset Selection for Specialization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916702864801792 |
|---|---|
| author | Hulkund, Neha Maalouf, Alaa Cai, Levi Yang, Daniel Wang, Tsun-Hsuan O'Neil, Abigail Haucke, Timm Mukherjee, Sandeep Ramaswamy, Vikram Shen, Judy Hansen Tseng, Gabriel Walmsley, Mike Rus, Daniela Goldberg, Ken Kerner, Hannah Chen, Irene Girdhar, Yogesh Beery, Sara |
| author_facet | Hulkund, Neha Maalouf, Alaa Cai, Levi Yang, Daniel Wang, Tsun-Hsuan O'Neil, Abigail Haucke, Timm Mukherjee, Sandeep Ramaswamy, Vikram Shen, Judy Hansen Tseng, Gabriel Walmsley, Mike Rus, Daniela Goldberg, Ken Kerner, Hannah Chen, Irene Girdhar, Yogesh Beery, Sara |
| contents | In many real-world machine learning (ML) applications (e.g. detecting broken bones in x-ray images, detecting species in camera traps), in practice models need to perform well on specific deployments (e.g. a specific hospital, a specific national park) rather than the domain broadly. However, deployments often have imbalanced, unique data distributions. Discrepancy between the training distribution and the deployment distribution can lead to suboptimal performance, highlighting the need to select deployment-specialized subsets from the available training data. We formalize dataset subset selection for specialization (DS3): given a training set drawn from a general distribution and a (potentially unlabeled) query set drawn from the desired deployment-specific distribution, the goal is to select a subset of the training data that optimizes deployment performance.
We introduce DataS^3; the first dataset and benchmark designed specifically for the DS3 problem. DataS^3 encompasses diverse real-world application domains, each with a set of distinct deployments to specialize in. We conduct a comprehensive study evaluating algorithms from various families--including coresets, data filtering, and data curation--on DataS^3, and find that general-distribution methods consistently fail on deployment-specific tasks. Additionally, we demonstrate the existence of manually curated (deployment-specific) expert subsets that outperform training on all available data with accuracy gains up to 51.3 percent. Our benchmark highlights the critical role of tailored dataset curation in enhancing performance and training efficiency on deployment-specific distributions, which we posit will only become more important as global, public datasets become available across domains and ML models are deployed in the real world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16277 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DataS^3: Dataset Subset Selection for Specialization Hulkund, Neha Maalouf, Alaa Cai, Levi Yang, Daniel Wang, Tsun-Hsuan O'Neil, Abigail Haucke, Timm Mukherjee, Sandeep Ramaswamy, Vikram Shen, Judy Hansen Tseng, Gabriel Walmsley, Mike Rus, Daniela Goldberg, Ken Kerner, Hannah Chen, Irene Girdhar, Yogesh Beery, Sara Machine Learning Artificial Intelligence In many real-world machine learning (ML) applications (e.g. detecting broken bones in x-ray images, detecting species in camera traps), in practice models need to perform well on specific deployments (e.g. a specific hospital, a specific national park) rather than the domain broadly. However, deployments often have imbalanced, unique data distributions. Discrepancy between the training distribution and the deployment distribution can lead to suboptimal performance, highlighting the need to select deployment-specialized subsets from the available training data. We formalize dataset subset selection for specialization (DS3): given a training set drawn from a general distribution and a (potentially unlabeled) query set drawn from the desired deployment-specific distribution, the goal is to select a subset of the training data that optimizes deployment performance. We introduce DataS^3; the first dataset and benchmark designed specifically for the DS3 problem. DataS^3 encompasses diverse real-world application domains, each with a set of distinct deployments to specialize in. We conduct a comprehensive study evaluating algorithms from various families--including coresets, data filtering, and data curation--on DataS^3, and find that general-distribution methods consistently fail on deployment-specific tasks. Additionally, we demonstrate the existence of manually curated (deployment-specific) expert subsets that outperform training on all available data with accuracy gains up to 51.3 percent. Our benchmark highlights the critical role of tailored dataset curation in enhancing performance and training efficiency on deployment-specific distributions, which we posit will only become more important as global, public datasets become available across domains and ML models are deployed in the real world. |
| title | DataS^3: Dataset Subset Selection for Specialization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2504.16277 |