Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910970693025792 |
|---|---|
| author | Hoang, Huy Mai, Tien Varakantham, Pradeep Verma, Tanvi |
| author_facet | Hoang, Huy Mai, Tien Varakantham, Pradeep Verma, Tanvi |
| contents | Offline imitation learning typically learns from expert and unlabeled demonstrations, yet often overlooks the valuable signal in explicitly undesirable behaviors. In this work, we study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations. We propose a novel formulation that optimizes a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data. Although the resulting objective is a DC (Difference-of-Convex) program, we prove that it becomes convex when expert demonstrations outweigh undesirable demonstrations, enabling a practical and stable non-adversarial training objective. Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework. Extensive experiments on standard offline imitation learning benchmarks demonstrate that our approach consistently outperforms state-of-the-art baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21182 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations Hoang, Huy Mai, Tien Varakantham, Pradeep Verma, Tanvi Machine Learning Artificial Intelligence Offline imitation learning typically learns from expert and unlabeled demonstrations, yet often overlooks the valuable signal in explicitly undesirable behaviors. In this work, we study offline imitation learning from contrasting behaviors, where the dataset contains both expert and undesirable demonstrations. We propose a novel formulation that optimizes a difference of KL divergences over the state-action visitation distributions of expert and undesirable (or bad) data. Although the resulting objective is a DC (Difference-of-Convex) program, we prove that it becomes convex when expert demonstrations outweigh undesirable demonstrations, enabling a practical and stable non-adversarial training objective. Our method avoids adversarial training and handles both positive and negative demonstrations in a unified framework. Extensive experiments on standard offline imitation learning benchmarks demonstrate that our approach consistently outperforms state-of-the-art baselines. |
| title | Learning What to Do and What Not To Do: Offline Imitation from Expert and Undesirable Demonstrations |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2505.21182 |