Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.17501 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908978312642560 |
|---|---|
| author | Xu, Ruiyao Parmar, Mihir Yang, Tiankai Hu, Zhengyu Zhao, Yue Ding, Kaize |
| author_facet | Xu, Ruiyao Parmar, Mihir Yang, Tiankai Hu, Zhengyu Zhao, Yue Ding, Kaize |
| contents | Learning from preference-based feedback has become an effective approach for aligning LLMs across diverse tasks. However, high-quality human-annotated preference data remains expensive and scarce. Existing methods address this challenge through either self-rewarding, which scales by using purely AI-generated labels but risks unreliability, or active learning, which ensures quality through oracle annotation but cannot fully leverage unlabeled data. In this paper, we present CoAct, a novel framework that synergistically combines self-rewarding and active learning through strategic human-AI collaboration. CoAct leverages self-consistency to identify both reliable self-labeled data and samples that require oracle verification. Additionally, oracle feedback guides the model to generate new instructions within its solvable capability. Evaluated on three reasoning benchmarks across two model families, CoAct achieves average improvements of +13.25% on GSM8K, +8.19% on MATH, and +13.16% on WebInstruct, consistently outperforming all baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_17501 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CoAct: Co-Active LLM Preference Learning with Human-AI Synergy Xu, Ruiyao Parmar, Mihir Yang, Tiankai Hu, Zhengyu Zhao, Yue Ding, Kaize Computation and Language Learning from preference-based feedback has become an effective approach for aligning LLMs across diverse tasks. However, high-quality human-annotated preference data remains expensive and scarce. Existing methods address this challenge through either self-rewarding, which scales by using purely AI-generated labels but risks unreliability, or active learning, which ensures quality through oracle annotation but cannot fully leverage unlabeled data. In this paper, we present CoAct, a novel framework that synergistically combines self-rewarding and active learning through strategic human-AI collaboration. CoAct leverages self-consistency to identify both reliable self-labeled data and samples that require oracle verification. Additionally, oracle feedback guides the model to generate new instructions within its solvable capability. Evaluated on three reasoning benchmarks across two model families, CoAct achieves average improvements of +13.25% on GSM8K, +8.19% on MATH, and +13.16% on WebInstruct, consistently outperforming all baselines. |
| title | CoAct: Co-Active LLM Preference Learning with Human-AI Synergy |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.17501 |