Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913535976538112 |
|---|---|
| author | Meng, Tao Mehrabi, Ninareh Goyal, Palash Ramakrishna, Anil Galstyan, Aram Zemel, Richard Chang, Kai-Wei Gupta, Rahul Peris, Charith |
| author_facet | Meng, Tao Mehrabi, Ninareh Goyal, Palash Ramakrishna, Anil Galstyan, Aram Zemel, Richard Chang, Kai-Wei Gupta, Rahul Peris, Charith |
| contents | We propose a constraint learning schema for fine-tuning Large Language Models (LLMs) with attribute control. Given a training corpus and control criteria formulated as a sequence-level constraint on model outputs, our method fine-tunes the LLM on the training corpus while enhancing constraint satisfaction with minimal impact on its utility and generation quality. Specifically, our approach regularizes the LLM training by penalizing the KL divergence between the desired output distribution, which satisfies the constraints, and the LLM's posterior. This regularization term can be approximated by an auxiliary model trained to decompose the sequence-level constraints into token-level guidance, allowing the term to be measured by a closed-form formulation. To further improve efficiency, we design a parallel scheme for concurrently updating both the LLM and the auxiliary model. We evaluate the empirical performance of our approach by controlling the toxicity when training an LLM. We show that our approach leads to an LLM that produces fewer inappropriate responses while achieving competitive performance on benchmarks and a toxicity detection task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_05559 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification Meng, Tao Mehrabi, Ninareh Goyal, Palash Ramakrishna, Anil Galstyan, Aram Zemel, Richard Chang, Kai-Wei Gupta, Rahul Peris, Charith Computation and Language We propose a constraint learning schema for fine-tuning Large Language Models (LLMs) with attribute control. Given a training corpus and control criteria formulated as a sequence-level constraint on model outputs, our method fine-tunes the LLM on the training corpus while enhancing constraint satisfaction with minimal impact on its utility and generation quality. Specifically, our approach regularizes the LLM training by penalizing the KL divergence between the desired output distribution, which satisfies the constraints, and the LLM's posterior. This regularization term can be approximated by an auxiliary model trained to decompose the sequence-level constraints into token-level guidance, allowing the term to be measured by a closed-form formulation. To further improve efficiency, we design a parallel scheme for concurrently updating both the LLM and the auxiliary model. We evaluate the empirical performance of our approach by controlling the toxicity when training an LLM. We show that our approach leads to an LLM that produces fewer inappropriate responses while achieving competitive performance on benchmarks and a toxicity detection task. |
| title | Attribute Controlled Fine-tuning for Large Language Models: A Case Study on Detoxification |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.05559 |