Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917285796511744 |
|---|---|
| author | Qian, Zhiqin Diaz, Ryan Seo, Sangwon Unhelkar, Vaibhav |
| author_facet | Qian, Zhiqin Diaz, Ryan Seo, Sangwon Unhelkar, Vaibhav |
| contents | When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behavior with human-provided specifications becomes critical for responsible AI deployment. Reward design provides a direct channel for such alignment by translating human expectations into reward functions that guide reinforcement learning (RL). However, existing methods are often too limited to capture nuanced human preferences that arise in long-horizon tasks. Hence, we introduce Hierarchical Reward Design from Language (HRDL): a problem formulation that extends classical reward design to encode richer behavioral specifications for hierarchical RL agents. We further propose Language to Hierarchical Rewards (L2HR) as a solution to HRDL. Experiments show that AI agents trained with rewards designed via L2HR not only complete tasks effectively but also better adhere to human specifications. Together, HRDL and L2HR advance the research on human-aligned AI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18582 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications Qian, Zhiqin Diaz, Ryan Seo, Sangwon Unhelkar, Vaibhav Artificial Intelligence Computation and Language Human-Computer Interaction Machine Learning I.2.11 When training artificial intelligence (AI) to perform tasks, humans often care not only about whether a task is completed but also how it is performed. As AI agents tackle increasingly complex tasks, aligning their behavior with human-provided specifications becomes critical for responsible AI deployment. Reward design provides a direct channel for such alignment by translating human expectations into reward functions that guide reinforcement learning (RL). However, existing methods are often too limited to capture nuanced human preferences that arise in long-horizon tasks. Hence, we introduce Hierarchical Reward Design from Language (HRDL): a problem formulation that extends classical reward design to encode richer behavioral specifications for hierarchical RL agents. We further propose Language to Hierarchical Rewards (L2HR) as a solution to HRDL. Experiments show that AI agents trained with rewards designed via L2HR not only complete tasks effectively but also better adhere to human specifications. Together, HRDL and L2HR advance the research on human-aligned AI agents. |
| title | Hierarchical Reward Design from Language: Enhancing Alignment of Agent Behavior with Human Specifications |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction Machine Learning I.2.11 |
| url | https://arxiv.org/abs/2602.18582 |