PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908660222918656 |
|---|---|
| author | Sun, Shengjie Lyu, Jiafei Liu, Runze Yan, Mengbei Liu, Bo Ye, Deheng Li, Xiu |
| author_facet | Sun, Shengjie Lyu, Jiafei Liu, Runze Yan, Mengbei Liu, Bo Ye, Deheng Li, Xiu |
| contents | Offline imitation learning (offline IL) enables training effective policies without requiring explicit reward annotations. Recent approaches attempt to estimate rewards for unlabeled datasets using a small set of expert demonstrations. However, these methods often assume that the similarity between a trajectory and an expert demonstration is positively correlated with the reward, which oversimplifies the underlying reward structure. We propose PROF, a novel framework that leverages large language models (LLMs) to generate and improve executable reward function codes from natural language descriptions and a single expert trajectory. We propose Reward Preference Ranking (RPR), a novel reward function quality assessment and ranking strategy without requiring environment interactions or RL training. RPR calculates the dominance scores of the reward functions, where higher scores indicate better alignment with expert preferences. By alternating between RPR and text-based gradient optimization, PROF fully automates the selection and refinement of optimal reward functions for downstream policy learning. Empirical results on D4RL demonstrate that PROF surpasses or matches recent strong baselines across numerous datasets and domains, highlighting the effectiveness of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_13765 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning Sun, Shengjie Lyu, Jiafei Liu, Runze Yan, Mengbei Liu, Bo Ye, Deheng Li, Xiu Machine Learning Artificial Intelligence Offline imitation learning (offline IL) enables training effective policies without requiring explicit reward annotations. Recent approaches attempt to estimate rewards for unlabeled datasets using a small set of expert demonstrations. However, these methods often assume that the similarity between a trajectory and an expert demonstration is positively correlated with the reward, which oversimplifies the underlying reward structure. We propose PROF, a novel framework that leverages large language models (LLMs) to generate and improve executable reward function codes from natural language descriptions and a single expert trajectory. We propose Reward Preference Ranking (RPR), a novel reward function quality assessment and ranking strategy without requiring environment interactions or RL training. RPR calculates the dominance scores of the reward functions, where higher scores indicate better alignment with expert preferences. By alternating between RPR and text-based gradient optimization, PROF fully automates the selection and refinement of optimal reward functions for downstream policy learning. Empirical results on D4RL demonstrate that PROF surpasses or matches recent strong baselines across numerous datasets and domains, highlighting the effectiveness of our approach. |
| title | PROF: An LLM-based Reward Code Preference Optimization Framework for Offline Imitation Learning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2511.13765 |