Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918318629191680 |
|---|---|
| author | Takakura, Shokichi Wachi, Akifumi Higuchi, Rei Miyaguchi, Kohei Suzuki, Taiji |
| author_facet | Takakura, Shokichi Wachi, Akifumi Higuchi, Rei Miyaguchi, Kohei Suzuki, Taiji |
| contents | Aligning large language models (LLMs) to diverse human preferences is fundamentally challenging since criteria can often conflict with each other. Inference-time alignment methods have recently gained popularity as they allow LLMs to be aligned to multiple criteria via different alignment algorithms at inference time. However, inference-time alignment is computationally expensive since it often requires multiple forward passes of the base model. In this work, we propose inference-aware meta-alignment (IAMA), a novel approach that enables LLMs to be aligned to multiple criteria with limited computational budget at inference time. IAMA trains a base model such that it can be effectively aligned to multiple tasks via different inference-time alignment algorithms. To solve the non-linear optimization problems involved in IAMA, we propose non-linear GRPO, which provably converges to the optimal solution in the space of probability measures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01603 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO Takakura, Shokichi Wachi, Akifumi Higuchi, Rei Miyaguchi, Kohei Suzuki, Taiji Machine Learning Aligning large language models (LLMs) to diverse human preferences is fundamentally challenging since criteria can often conflict with each other. Inference-time alignment methods have recently gained popularity as they allow LLMs to be aligned to multiple criteria via different alignment algorithms at inference time. However, inference-time alignment is computationally expensive since it often requires multiple forward passes of the base model. In this work, we propose inference-aware meta-alignment (IAMA), a novel approach that enables LLMs to be aligned to multiple criteria with limited computational budget at inference time. IAMA trains a base model such that it can be effectively aligned to multiple tasks via different inference-time alignment algorithms. To solve the non-linear optimization problems involved in IAMA, we propose non-linear GRPO, which provably converges to the optimal solution in the space of probability measures. |
| title | Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.01603 |