The Chicken and Egg Dilemma: Co-optimizing Data and Model Configurations for LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911456457392128 |
|---|---|
| author | Chen, Zhiliang Leong, Alfred Wei Lun Ong, Shao Yong Hemachandra, Apivich Lau, Gregory Kang Ruey Foo, Chuan-Sheng Liu, Zhengyuan Chen, Nancy F. Low, Bryan Kian Hsiang |
| author_facet | Chen, Zhiliang Leong, Alfred Wei Lun Ong, Shao Yong Hemachandra, Apivich Lau, Gregory Kang Ruey Foo, Chuan-Sheng Liu, Zhengyuan Chen, Nancy F. Low, Bryan Kian Hsiang |
| contents | Co-optimizing data and model configurations for training LLMs presents a classic chicken-and-egg dilemma: The best training data configuration (e.g., data mixture) for a downstream task depends on the chosen model configuration (e.g., model architecture), and vice versa. However, jointly optimizing both data and model configurations is often deemed intractable, and existing methods focus on either data or model optimization without considering their interaction. We introduce JoBS, an approach that uses a scaling-law-inspired performance predictor to aid Bayesian optimization (BO) in jointly optimizing LLM training data and model configurations efficiently. JoBS allocates a portion of the optimization budget to learn an LLM performance predictor that predicts how promising a training configuration is from a small number of training steps. The remaining budget is used to perform BO entirely with the predictor, effectively amortizing the cost of running full-training runs. We study JoBS's average regret and devise the optimal budget allocation to minimize regret. JoBS outperforms existing multi-fidelity BO baselines, as well as data and model optimization approaches across diverse LLM tasks under the same optimization budget. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_08351 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | The Chicken and Egg Dilemma: Co-optimizing Data and Model Configurations for LLMs Chen, Zhiliang Leong, Alfred Wei Lun Ong, Shao Yong Hemachandra, Apivich Lau, Gregory Kang Ruey Foo, Chuan-Sheng Liu, Zhengyuan Chen, Nancy F. Low, Bryan Kian Hsiang Machine Learning Artificial Intelligence Co-optimizing data and model configurations for training LLMs presents a classic chicken-and-egg dilemma: The best training data configuration (e.g., data mixture) for a downstream task depends on the chosen model configuration (e.g., model architecture), and vice versa. However, jointly optimizing both data and model configurations is often deemed intractable, and existing methods focus on either data or model optimization without considering their interaction. We introduce JoBS, an approach that uses a scaling-law-inspired performance predictor to aid Bayesian optimization (BO) in jointly optimizing LLM training data and model configurations efficiently. JoBS allocates a portion of the optimization budget to learn an LLM performance predictor that predicts how promising a training configuration is from a small number of training steps. The remaining budget is used to perform BO entirely with the predictor, effectively amortizing the cost of running full-training runs. We study JoBS's average regret and devise the optimal budget allocation to minimize regret. JoBS outperforms existing multi-fidelity BO baselines, as well as data and model optimization approaches across diverse LLM tasks under the same optimization budget. |
| title | The Chicken and Egg Dilemma: Co-optimizing Data and Model Configurations for LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.08351 |