NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909302911926272 |
|---|---|
| author | Shen, Gerald Wang, Zhilin Delalleau, Olivier Zeng, Jiaqi Dong, Yi Egert, Daniel Sun, Shengyang Zhang, Jimmy Jain, Sahil Taghibakhshi, Ali Ausin, Markel Sanz Aithal, Ashwath Kuchaiev, Oleksii |
| author_facet | Shen, Gerald Wang, Zhilin Delalleau, Olivier Zeng, Jiaqi Dong, Yi Egert, Daniel Sun, Shengyang Zhang, Jimmy Jain, Sahil Taghibakhshi, Ali Ausin, Markel Sanz Aithal, Ashwath Kuchaiev, Oleksii |
| contents | Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to a thousand GPUs for training the largest open-source LLMs such as Nemotron 4 340B and Llama 3.1 405B. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01481 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment Shen, Gerald Wang, Zhilin Delalleau, Olivier Zeng, Jiaqi Dong, Yi Egert, Daniel Sun, Shengyang Zhang, Jimmy Jain, Sahil Taghibakhshi, Ali Ausin, Markel Sanz Aithal, Ashwath Kuchaiev, Oleksii Computation and Language Artificial Intelligence Machine Learning Aligning Large Language Models (LLMs) with human values and preferences is essential for making them helpful and safe. However, building efficient tools to perform alignment can be challenging, especially for the largest and most competent LLMs which often contain tens or hundreds of billions of parameters. We create NeMo-Aligner, a toolkit for model alignment that can efficiently scale to a thousand GPUs for training the largest open-source LLMs such as Nemotron 4 340B and Llama 3.1 405B. NeMo-Aligner comes with highly optimized and scalable implementations for major paradigms of model alignment such as: Reinforcement Learning from Human Feedback (RLHF), Direct Preference Optimization (DPO), SteerLM, and Self-Play Fine-Tuning (SPIN). Additionally, our toolkit supports running most of the alignment techniques in a Parameter Efficient Fine-Tuning (PEFT) setting. NeMo-Aligner is designed for extensibility, allowing support for other alignment techniques with minimal effort. It is open-sourced with Apache 2.0 License and we invite community contributions at https://github.com/NVIDIA/NeMo-Aligner |
| title | NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2405.01481 |