Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917491448479744 |
|---|---|
| author | Osman, Asim Abramowitz, Sasha Bergh, Mark Sob, Ulrich Armel Mbou de Kock, Ruan John Mahjoub, Omayma Hidaoui, Oussama De Nicola, Noah Fokam, Arnol Manuel Chalumeau, Felix Rajaonarivonivelomanantsoa, Daniel Singh, Siddarth Shabe, Refiloe Formanek, Juan Claude Toit, Simon Verster Du Pretorius, Arnu |
| author_facet | Osman, Asim Abramowitz, Sasha Bergh, Mark Sob, Ulrich Armel Mbou de Kock, Ruan John Mahjoub, Omayma Hidaoui, Oussama De Nicola, Noah Fokam, Arnol Manuel Chalumeau, Felix Rajaonarivonivelomanantsoa, Daniel Singh, Siddarth Shabe, Refiloe Formanek, Juan Claude Toit, Simon Verster Du Pretorius, Arnu |
| contents | Contrastive reinforcement learning (CRL) learns goal-conditioned Q-values through a contrastive objective over state-action and goal representations, removing the need for hand-crafted reward functions. Despite impressive success in achieving viable self-supervised learning in RL, all existing CRL algorithms rely on off-policy optimisation and are mostly constrained to continuous action spaces, with little research invested in discrete environments. This leaves CRL disconnected from widely used and effective, modern on-policy training pipelines adopted across both single-agent and multi-agent RL in continuous and discrete environments. To establish a first connection, we introduce Contrastive Proximal Policy Optimisation (CPPO). CPPO is an on-policy contrastive RL algorithm that derives policy advantages directly from contrastive Q-values and optimises them via the standard PPO objective, without requiring a reward function or a replay buffer. We evaluate CPPO across continuous and discrete, single-agent and cooperative multi-agent tasks. Whilst the existence of an on-policy approach is inherently useful, we observe that \textbf{CPPO not only significantly outperforms the previous CRL baselines in 14 out of 18 tasks, but also matches or exceeds PPO's performance, which uses hand-crafted dense rewards, in 12 out of the 18 tasks tested.} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13554 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation Osman, Asim Abramowitz, Sasha Bergh, Mark Sob, Ulrich Armel Mbou de Kock, Ruan John Mahjoub, Omayma Hidaoui, Oussama De Nicola, Noah Fokam, Arnol Manuel Chalumeau, Felix Rajaonarivonivelomanantsoa, Daniel Singh, Siddarth Shabe, Refiloe Formanek, Juan Claude Toit, Simon Verster Du Pretorius, Arnu Machine Learning Artificial Intelligence 68T05 (Primary) 68T07, 90C4 (Secondary) I.2.6; I.2.8 Contrastive reinforcement learning (CRL) learns goal-conditioned Q-values through a contrastive objective over state-action and goal representations, removing the need for hand-crafted reward functions. Despite impressive success in achieving viable self-supervised learning in RL, all existing CRL algorithms rely on off-policy optimisation and are mostly constrained to continuous action spaces, with little research invested in discrete environments. This leaves CRL disconnected from widely used and effective, modern on-policy training pipelines adopted across both single-agent and multi-agent RL in continuous and discrete environments. To establish a first connection, we introduce Contrastive Proximal Policy Optimisation (CPPO). CPPO is an on-policy contrastive RL algorithm that derives policy advantages directly from contrastive Q-values and optimises them via the standard PPO objective, without requiring a reward function or a replay buffer. We evaluate CPPO across continuous and discrete, single-agent and cooperative multi-agent tasks. Whilst the existence of an on-policy approach is inherently useful, we observe that \textbf{CPPO not only significantly outperforms the previous CRL baselines in 14 out of 18 tasks, but also matches or exceeds PPO's performance, which uses hand-crafted dense rewards, in 12 out of the 18 tasks tested.} |
| title | Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation |
| topic | Machine Learning Artificial Intelligence 68T05 (Primary) 68T07, 90C4 (Secondary) I.2.6; I.2.8 |
| url | https://arxiv.org/abs/2605.13554 |