Group-Aware Reinforcement Learning for Output Diversity in Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914160249405440 |
|---|---|
| author | Anschel, Oron Shoshan, Alon Botach, Adam Hakimi, Shunit Haviv Gendler, Asaf Baruch, Emanuel Ben Bhonker, Nadav Kviatkovsky, Igor Aggarwal, Manoj Medioni, Gerard |
| author_facet | Anschel, Oron Shoshan, Alon Botach, Adam Hakimi, Shunit Haviv Gendler, Asaf Baruch, Emanuel Ben Bhonker, Nadav Kviatkovsky, Igor Aggarwal, Manoj Medioni, Gerard |
| contents | Large Language Models (LLMs) often suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist, limiting their diversity across a wide range of tasks. We introduce Group-Aware Policy Optimization (GAPO), a simple extension of the recent and popular Group Relative Policy Optimization (GRPO) that computes rewards over the group as a whole. GAPO enables learning from the group-level properties such as diversity and coverage. We demonstrate GAPO using a frequency-aware reward function that encourages uniform sampling over valid LLM completions, and show that GAPO-trained models produce valid and more diverse model responses. Beyond this setup, GAPO generalizes to open-ended prompts and improves response diversity without compromising accuracy on standard LLM benchmarks (GSM8K, MATH, HumanEval, MMLU-Pro). Our code will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12596 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Group-Aware Reinforcement Learning for Output Diversity in Large Language Models Anschel, Oron Shoshan, Alon Botach, Adam Hakimi, Shunit Haviv Gendler, Asaf Baruch, Emanuel Ben Bhonker, Nadav Kviatkovsky, Igor Aggarwal, Manoj Medioni, Gerard Computation and Language Artificial Intelligence Machine Learning Large Language Models (LLMs) often suffer from mode collapse, repeatedly generating the same few completions even when many valid answers exist, limiting their diversity across a wide range of tasks. We introduce Group-Aware Policy Optimization (GAPO), a simple extension of the recent and popular Group Relative Policy Optimization (GRPO) that computes rewards over the group as a whole. GAPO enables learning from the group-level properties such as diversity and coverage. We demonstrate GAPO using a frequency-aware reward function that encourages uniform sampling over valid LLM completions, and show that GAPO-trained models produce valid and more diverse model responses. Beyond this setup, GAPO generalizes to open-ended prompts and improves response diversity without compromising accuracy on standard LLM benchmarks (GSM8K, MATH, HumanEval, MMLU-Pro). Our code will be made publicly available. |
| title | Group-Aware Reinforcement Learning for Output Diversity in Large Language Models |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.12596 |