Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911960329617408 |
|---|---|
| author | Lee, Janghwan Park, Seongmin Hong, Sukjin Kim, Minsoo Chang, Du-Seong Choi, Jungwook |
| author_facet | Lee, Janghwan Park, Seongmin Hong, Sukjin Kim, Minsoo Chang, Du-Seong Choi, Jungwook |
| contents | The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03051 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment Lee, Janghwan Park, Seongmin Hong, Sukjin Kim, Minsoo Chang, Du-Seong Choi, Jungwook Computation and Language The rapid advancement of large language models (LLMs) has facilitated their transformation into conversational chatbots that can grasp contextual nuances and generate pertinent sentences, closely mirroring human values through advanced techniques such as instruction tuning and reinforcement learning from human feedback (RLHF). However, the computational efficiency required for LLMs, achieved through techniques like post-training quantization (PTQ), presents challenges such as token-flipping that can impair chatbot performance. In response, we propose a novel preference alignment approach, quantization-aware direct preference optimization (QDPO), that aligns quantized LLMs with their full-precision counterparts, improving conversational abilities. Evaluated on two instruction-tuned LLMs in various languages, QDPO demonstrated superior performance in improving conversational abilities compared to established PTQ and knowledge-distillation fine-tuning techniques, marking a significant step forward in the development of efficient and effective conversational LLMs. |
| title | Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.03051 |