Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929708471418880 |
|---|---|
| author | Wang, Huanqian Yue, Yang Lu, Rui Shi, Jingxin Zhao, Andrew Wang, Shenzhi Song, Shiji Huang, Gao |
| author_facet | Wang, Huanqian Yue, Yang Lu, Rui Shi, Jingxin Zhao, Andrew Wang, Shenzhi Song, Shiji Huang, Gao |
| contents | Large Language Models (LLMs) have demonstrated great potential as generalist assistants, showcasing powerful task understanding and problem-solving capabilities. To deploy LLMs as AI assistants, it is crucial that these models exhibit desirable behavioral traits, such as non-toxicity and resilience against jailbreak attempts. Current approaches for detoxification or preventing jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), which requires finetuning billions of parameters through gradient descent with substantial computational cost. Furthermore, models modified through SFT and RLHF may deviate from the pretrained models, potentially leading to a degradation in foundational LLM capabilities. In this paper, we observe that surprisingly, directly editing a small subset of parameters can effectively modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreaking, with only inference-level computational resources. Experiments demonstrate that in the detoxification task, our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen, while maintaining the LLM's general capabilities in areas such as common sense, question answering, and mathematics |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08770 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing Wang, Huanqian Yue, Yang Lu, Rui Shi, Jingxin Zhao, Andrew Wang, Shenzhi Song, Shiji Huang, Gao Artificial Intelligence 68T50 (Primary) 68T07, 62M45 (Secondary) I.2.7 Large Language Models (LLMs) have demonstrated great potential as generalist assistants, showcasing powerful task understanding and problem-solving capabilities. To deploy LLMs as AI assistants, it is crucial that these models exhibit desirable behavioral traits, such as non-toxicity and resilience against jailbreak attempts. Current approaches for detoxification or preventing jailbreaking usually involve Supervised Fine-Tuning (SFT) or Reinforcement Learning from Human Feedback (RLHF), which requires finetuning billions of parameters through gradient descent with substantial computational cost. Furthermore, models modified through SFT and RLHF may deviate from the pretrained models, potentially leading to a degradation in foundational LLM capabilities. In this paper, we observe that surprisingly, directly editing a small subset of parameters can effectively modulate specific behaviors of LLMs, such as detoxification and resistance to jailbreaking, with only inference-level computational resources. Experiments demonstrate that in the detoxification task, our approach achieves reductions of up to 90.0% in toxicity on the RealToxicityPrompts dataset and 49.2% on ToxiGen, while maintaining the LLM's general capabilities in areas such as common sense, question answering, and mathematics |
| title | Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing |
| topic | Artificial Intelligence 68T50 (Primary) 68T07, 62M45 (Secondary) I.2.7 |
| url | https://arxiv.org/abs/2407.08770 |