From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912643969712128 |
|---|---|
| author | Benechehab, Abdelhakim Singer, Gabriel Léger, Corentin Hili, Youssef Attia El Paolo, Giuseppe Thomas, Albert Filippone, Maurizio Kégl, Balázs |
| author_facet | Benechehab, Abdelhakim Singer, Gabriel Léger, Corentin Hili, Youssef Attia El Paolo, Giuseppe Thomas, Albert Filippone, Maurizio Kégl, Balázs |
| contents | Generative models form the backbone of modern machine learning, underpinning state-of-the-art systems in text, vision, and multimodal applications. While Maximum Likelihood Estimation has traditionally served as the dominant training paradigm, recent work have highlighted its limitations, particularly in generalization and susceptibility to catastrophic forgetting compared to Reinforcement Learning techniques, such as Policy Gradient methods. However, these approaches depend on explicit reward signals, which are often unavailable in practice, leaving open the fundamental problem of how to align generative models when only high-quality datasets are accessible. In this work, we address this challenge via a Bilevel Optimization framework, where the reward function is treated as the optimization variable of an outer-level problem, while a policy gradient objective defines the inner-level. We then conduct a theoretical analysis of this optimization problem in a tractable setting and extract insights that, as we demonstrate, generalize to applications such as tabular classification and model-based reinforcement learning. We release the code at https://github.com/abenechehab/nll_to_po . |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_07624 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation Benechehab, Abdelhakim Singer, Gabriel Léger, Corentin Hili, Youssef Attia El Paolo, Giuseppe Thomas, Albert Filippone, Maurizio Kégl, Balázs Machine Learning Generative models form the backbone of modern machine learning, underpinning state-of-the-art systems in text, vision, and multimodal applications. While Maximum Likelihood Estimation has traditionally served as the dominant training paradigm, recent work have highlighted its limitations, particularly in generalization and susceptibility to catastrophic forgetting compared to Reinforcement Learning techniques, such as Policy Gradient methods. However, these approaches depend on explicit reward signals, which are often unavailable in practice, leaving open the fundamental problem of how to align generative models when only high-quality datasets are accessible. In this work, we address this challenge via a Bilevel Optimization framework, where the reward function is treated as the optimization variable of an outer-level problem, while a policy gradient objective defines the inner-level. We then conduct a theoretical analysis of this optimization problem in a tractable setting and extract insights that, as we demonstrate, generalize to applications such as tabular classification and model-based reinforcement learning. We release the code at https://github.com/abenechehab/nll_to_po . |
| title | From Data to Rewards: a Bilevel Optimization Perspective on Maximum Likelihood Estimation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2510.07624 |