Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.10042 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911312606396416 |
|---|---|
| author | Lee, Jongmin Sun, Meiqi Abbeel, Pieter |
| author_facet | Lee, Jongmin Sun, Meiqi Abbeel, Pieter |
| contents | In the unsupervised pre-training for reinforcement learning, the agent aims to learn a prior policy for downstream tasks without relying on task-specific reward functions. We focus on state entropy maximization (SEM), where the goal is to learn a policy that maximizes the entropy of the state stationary distribution. In this paper, we introduce SEMDICE, a principled off-policy algorithm that computes an SEM policy from an arbitrary off-policy dataset, which optimizes the policy directly within the space of stationary distributions. SEMDICE computes a single, stationary Markov state-entropy-maximizing policy from an arbitrary off-policy dataset. Experimental results demonstrate that SEMDICE outperforms baseline algorithms in maximizing state entropy while achieving the best adaptation efficiency for downstream tasks among SEM-based unsupervised RL pre-training methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_10042 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation Lee, Jongmin Sun, Meiqi Abbeel, Pieter Machine Learning In the unsupervised pre-training for reinforcement learning, the agent aims to learn a prior policy for downstream tasks without relying on task-specific reward functions. We focus on state entropy maximization (SEM), where the goal is to learn a policy that maximizes the entropy of the state stationary distribution. In this paper, we introduce SEMDICE, a principled off-policy algorithm that computes an SEM policy from an arbitrary off-policy dataset, which optimizes the policy directly within the space of stationary distributions. SEMDICE computes a single, stationary Markov state-entropy-maximizing policy from an arbitrary off-policy dataset. Experimental results demonstrate that SEMDICE outperforms baseline algorithms in maximizing state entropy while achieving the best adaptation efficiency for downstream tasks among SEM-based unsupervised RL pre-training methods. |
| title | SEMDICE: Off-policy State Entropy Maximization via Stationary Distribution Correction Estimation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2512.10042 |