Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912539496939520 |
|---|---|
| author | Zhang, Yang Yu, Yu Tang, Bo Zhu, Yu Sun, Chuxiong Wei, Wenqiang Hu, Jie Xie, Zipeng Li, Zhiyu Xiong, Feiyu Chung, Edward |
| author_facet | Zhang, Yang Yu, Yu Tang, Bo Zhu, Yu Sun, Chuxiong Wei, Wenqiang Hu, Jie Xie, Zipeng Li, Zhiyu Xiong, Feiyu Chung, Edward |
| contents | With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19743 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models Zhang, Yang Yu, Yu Tang, Bo Zhu, Yu Sun, Chuxiong Wei, Wenqiang Hu, Jie Xie, Zipeng Li, Zhiyu Xiong, Feiyu Chung, Edward Computation and Language Machine Learning With the rapid development of Large Language Models (LLMs), aligning these models with human preferences and values is critical to ensuring ethical and safe applications. However, existing alignment techniques such as RLHF or DPO often require direct fine-tuning on LLMs with billions of parameters, resulting in substantial computational costs and inefficiencies. To address this, we propose Micro token-level Accept-Reject Aligning (MARA) approach designed to operate independently of the language models. MARA simplifies the alignment process by decomposing sentence-level preference learning into token-level binary classification, where a compact three-layer fully-connected network determines whether candidate tokens are "Accepted" or "Rejected" as part of the response. Extensive experiments across seven different LLMs and three open-source datasets show that MARA achieves significant improvements in alignment performance while reducing computational costs. The source code and implementation details are publicly available at https://github.com/IAAR-Shanghai/MARA, and the trained models are released at https://huggingface.co/IAAR-Shanghai/MARA_AGENTS. |
| title | Token-level Accept or Reject: A Micro Alignment Approach for Large Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2505.19743 |