From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916067979296768 |
|---|---|
| author | Gumbsch, Christian Barcellona, Leonardo Schünemann, Lennard Karageorgis, Platon Zadaianchuk, Andrii Wang, Zehao Zakharov, Sergey Despinoy, Fabien Aljundi, Rahaf Gavves, Efstratios |
| author_facet | Gumbsch, Christian Barcellona, Leonardo Schünemann, Lennard Karageorgis, Platon Zadaianchuk, Andrii Wang, Zehao Zakharov, Sergey Despinoy, Fabien Aljundi, Rahaf Gavves, Efstratios |
| contents | Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior policy training. We propose Demo2Reward a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3-10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_00083 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models Gumbsch, Christian Barcellona, Leonardo Schünemann, Lennard Karageorgis, Platon Zadaianchuk, Andrii Wang, Zehao Zakharov, Sergey Despinoy, Fabien Aljundi, Rahaf Gavves, Efstratios Machine Learning Artificial Intelligence Robotics Reinforcement learning relies on accurate reward functions, which are often hand-crafted or even unavailable in real-world applications, such as robotics. Recent work has explored the zero-shot reasoning capabilities of pre-trained Vision-Language Models (VLMs) as reward models. However, without careful prompt engineering, these approaches tend to produce suboptimal rewards, where false positive predictions can severely degrade downstream policy learning. In robotics, limited datasets comprising expert demonstrations are often collected to bootstrap policy learning. This scenario provides an opportunity to optimize a reward model prior policy training. We propose Demo2Reward a test-time adaptation technique to optimize the language instruction of a reward model based on a few demonstrations (3-10 trajectories) to reduce false positives while preserving true positives. Crucially, this requires no additional model training or computation resources during policy learning. We show that Demo2Reward consistently outperforms existing zero- and few-shot VLM reward models across a range of simulated robotic tasks and policy backbones. Finally, we demonstrate that Demo2Reward effectively transfers to a real-world robotic learning scenario, enabling policy learning without manually engineering a reward function. |
| title | From Demonstrations to Rewards: Test-Time Prompt Optimization for VLM Reward Models |
| topic | Machine Learning Artificial Intelligence Robotics |
| url | https://arxiv.org/abs/2606.00083 |