PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913899336433664 |
|---|---|
| author | Zhang, Yizhen Ding, Yang Zhang, Shuoshuo Zhang, Xinchen Li, Haoling Li, Zhong-zhi Wang, Peijie Wu, Jie Ji, Lei Shen, Yelong Yang, Yujiu Gong, Yeyun |
| author_facet | Zhang, Yizhen Ding, Yang Zhang, Shuoshuo Zhang, Xinchen Li, Haoling Li, Zhong-zhi Wang, Peijie Wu, Jie Ji, Lei Shen, Yelong Yang, Yujiu Gong, Yeyun |
| contents | Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-R1, recent emerging research has begun exploring the use of reinforcement learning (RL) to enhance vision-language models (VLMs) for multimodal reasoning tasks. However, most existing multimodal reinforcement learning approaches remain limited to spatial reasoning within single-image contexts, yet still struggle to generalize to more complex and real-world scenarios involving multi-image positional reasoning, where understanding the relationships across images is crucial. To address this challenge, we propose a general reinforcement learning approach PeRL tailored for interleaved multimodal tasks, and a multi-stage strategy designed to enhance the exploration-exploitation trade-off, thereby improving learning efficiency and task performance. Specifically, we introduce permutation of image sequences to simulate varied positional relationships to explore more spatial and positional diversity. Furthermore, we design a rollout filtering mechanism for resampling to focus on trajectories that contribute most to learning optimal behaviors to exploit learned policies effectively. We evaluate our model on 5 widely-used multi-image benchmarks and 3 single-image benchmarks. Our experiments confirm that PeRL trained model consistently surpasses R1-related and interleaved VLM baselines by a large margin, achieving state-of-the-art performance on multi-image benchmarks, while preserving comparable performance on single-image tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14907 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning Zhang, Yizhen Ding, Yang Zhang, Shuoshuo Zhang, Xinchen Li, Haoling Li, Zhong-zhi Wang, Peijie Wu, Jie Ji, Lei Shen, Yelong Yang, Yujiu Gong, Yeyun Computer Vision and Pattern Recognition Artificial Intelligence Inspired by the impressive reasoning capabilities demonstrated by reinforcement learning approaches like DeepSeek-R1, recent emerging research has begun exploring the use of reinforcement learning (RL) to enhance vision-language models (VLMs) for multimodal reasoning tasks. However, most existing multimodal reinforcement learning approaches remain limited to spatial reasoning within single-image contexts, yet still struggle to generalize to more complex and real-world scenarios involving multi-image positional reasoning, where understanding the relationships across images is crucial. To address this challenge, we propose a general reinforcement learning approach PeRL tailored for interleaved multimodal tasks, and a multi-stage strategy designed to enhance the exploration-exploitation trade-off, thereby improving learning efficiency and task performance. Specifically, we introduce permutation of image sequences to simulate varied positional relationships to explore more spatial and positional diversity. Furthermore, we design a rollout filtering mechanism for resampling to focus on trajectories that contribute most to learning optimal behaviors to exploit learned policies effectively. We evaluate our model on 5 widely-used multi-image benchmarks and 3 single-image benchmarks. Our experiments confirm that PeRL trained model consistently surpasses R1-related and interleaved VLM baselines by a large margin, achieving state-of-the-art performance on multi-image benchmarks, while preserving comparable performance on single-image tasks. |
| title | PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2506.14907 |