Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916770055454720 |
|---|---|
| author | Ren, Qingyu Zeng, Jie He, Qianyu Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei |
| author_facet | Ren, Qingyu Zeng, Jie He, Qianyu Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei |
| contents | It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, it is an unexplored area to enhance LLMs' ability to follow soft constraints. To bridge the gap, we initially design a pipeline to construct datasets with high-quality outputs automatically. Additionally, to fully utilize the positive and negative samples generated during the data construction process, we choose Direct Preference Optimization (DPO) as the training method. Furthermore, taking into account the difficulty of soft constraints indicated by the number of constraints, we design a curriculum learning training paradigm based on the constraint quantity. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements.The datasets and code are publicly available at https://github.com/Rainier-rq/FollowSoftConstraint. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_04945 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models Ren, Qingyu Zeng, Jie He, Qianyu Liang, Jiaqing Xiao, Yanghua Zhou, Weikang Sun, Zeye Yu, Fei Computation and Language Artificial Intelligence It is crucial for large language models (LLMs) to follow instructions that involve multiple constraints. However, it is an unexplored area to enhance LLMs' ability to follow soft constraints. To bridge the gap, we initially design a pipeline to construct datasets with high-quality outputs automatically. Additionally, to fully utilize the positive and negative samples generated during the data construction process, we choose Direct Preference Optimization (DPO) as the training method. Furthermore, taking into account the difficulty of soft constraints indicated by the number of constraints, we design a curriculum learning training paradigm based on the constraint quantity. We experimentally evaluate the effectiveness of our methods in improving LLMs' soft constraint following ability and analyze the factors driving the improvements.The datasets and code are publicly available at https://github.com/Rainier-rq/FollowSoftConstraint. |
| title | Step-by-Step Mastery: Enhancing Soft Constraint Following Ability of Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.04945 |