Optimizing Language Models for Inference Time Objectives using Reinforcement Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918126193475584 |
|---|---|
| author | Tang, Yunhao Zheng, Kunhao Synnaeve, Gabriel Munos, Rémi |
| author_facet | Tang, Yunhao Zheng, Kunhao Synnaeve, Gabriel Munos, Rémi |
| contents | In this work, we investigate the merits of explicitly optimizing for inference time algorithmic performance during model training. We show how optimizing for inference time performance can improve overall model efficacy. We consider generic inference time objectives with $k$ samples, with a focus on pass@$k$ and majority voting as two main applications. With language model training on reasoning datasets, we showcase the performance trade-off enabled by training with such objectives. When training on code generation tasks, we show that the approach significantly improves pass@$k$ objectives compared to the baseline method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_19595 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimizing Language Models for Inference Time Objectives using Reinforcement Learning Tang, Yunhao Zheng, Kunhao Synnaeve, Gabriel Munos, Rémi Machine Learning In this work, we investigate the merits of explicitly optimizing for inference time algorithmic performance during model training. We show how optimizing for inference time performance can improve overall model efficacy. We consider generic inference time objectives with $k$ samples, with a focus on pass@$k$ and majority voting as two main applications. With language model training on reasoning datasets, we showcase the performance trade-off enabled by training with such objectives. When training on code generation tasks, we show that the approach significantly improves pass@$k$ objectives compared to the baseline method. |
| title | Optimizing Language Models for Inference Time Objectives using Reinforcement Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.19595 |