MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910121716613120 |
|---|---|
| author | Zhao, Yang Wang, Hepeng Ding, Xiao Ouyang, Yangou Cai, Bibo Xiong, Kai Gao, Jinglong Sun, Zhouhao Du, Li Qin, Bing Liu, Ting |
| author_facet | Zhao, Yang Wang, Hepeng Ding, Xiao Ouyang, Yangou Cai, Bibo Xiong, Kai Gao, Jinglong Sun, Zhouhao Du, Li Qin, Bing Liu, Ting |
| contents | Group-Relative Policy Optimization (GRPO) has emerged as an efficient paradigm for aligning Large Language Models (LLMs), yet its efficacy is primarily confined to domains with verifiable ground truths. Extending GRPO to open-domain settings remains a critical challenge, as unconstrained generation entails multi-faceted and often conflicting objectives - such as creativity versus factuality - where rigid, static reward scalarization is inherently suboptimal. To address this, we propose MAESTRO (Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization), which introduces a meta-cognitive orchestration layer that treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck to perceive task-specific priorities. We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal. Across seven benchmarks, MAESTRO consistently outperforms single-reward and static multi-objective baselines, while preserving the efficiency advantages of GRPO, and in some settings even reducing redundant generation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_07208 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization Zhao, Yang Wang, Hepeng Ding, Xiao Ouyang, Yangou Cai, Bibo Xiong, Kai Gao, Jinglong Sun, Zhouhao Du, Li Qin, Bing Liu, Ting Machine Learning Computation and Language Group-Relative Policy Optimization (GRPO) has emerged as an efficient paradigm for aligning Large Language Models (LLMs), yet its efficacy is primarily confined to domains with verifiable ground truths. Extending GRPO to open-domain settings remains a critical challenge, as unconstrained generation entails multi-faceted and often conflicting objectives - such as creativity versus factuality - where rigid, static reward scalarization is inherently suboptimal. To address this, we propose MAESTRO (Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization), which introduces a meta-cognitive orchestration layer that treats reward scalarization as a dynamic latent policy, leveraging the model's terminal hidden states as a semantic bottleneck to perceive task-specific priorities. We formulate this as a contextual bandit problem within a bi-level optimization framework, where a lightweight Conductor network co-evolves with the policy by utilizing group-relative advantages as a meta-reward signal. Across seven benchmarks, MAESTRO consistently outperforms single-reward and static multi-objective baselines, while preserving the efficiency advantages of GRPO, and in some settings even reducing redundant generation. |
| title | MAESTRO: Meta-learning Adaptive Estimation of Scalarization Trade-offs for Reward Optimization |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.07208 |