Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916591278489600 |
|---|---|
| author | Lee, JoonHo Woo, Jae Oh Seok, Juree Hassanzadeh, Parisa Jang, Wooseok Son, JuYoun Didari, Sima Gutow, Baruch Hao, Heng Moon, Hankyu Hu, Wenjun Kwon, Yeong-Dae Lee, Taehee Min, Seungjai |
| author_facet | Lee, JoonHo Woo, Jae Oh Seok, Juree Hassanzadeh, Parisa Jang, Wooseok Son, JuYoun Didari, Sima Gutow, Baruch Hao, Heng Moon, Hankyu Hu, Wenjun Kwon, Yeong-Dae Lee, Taehee Min, Seungjai |
| contents | Assessing response quality to instructions in language models is vital but challenging due to the complexity of human language across different contexts. This complexity often results in ambiguous or inconsistent interpretations, making accurate assessment difficult. To address this issue, we propose a novel Uncertainty-aware Reward Model (URM) that introduces a robust uncertainty estimation for the quality of paired responses based on Bayesian approximation. Trained with preference datasets, our uncertainty-enabled proxy not only scores rewards for responses but also evaluates their inherent uncertainty. Empirical results demonstrate significant benefits of incorporating the proposed proxy into language model training. Our method boosts the instruction following capability of language models by refining data curation for training and improving policy optimization objectives, thereby surpassing existing methods by a large margin on benchmarks such as Vicuna and MT-bench. These findings highlight that our proposed approach substantially advances language model training and paves a new way of harnessing uncertainty within language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_06424 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation Lee, JoonHo Woo, Jae Oh Seok, Juree Hassanzadeh, Parisa Jang, Wooseok Son, JuYoun Didari, Sima Gutow, Baruch Hao, Heng Moon, Hankyu Hu, Wenjun Kwon, Yeong-Dae Lee, Taehee Min, Seungjai Computation and Language Artificial Intelligence Machine Learning Assessing response quality to instructions in language models is vital but challenging due to the complexity of human language across different contexts. This complexity often results in ambiguous or inconsistent interpretations, making accurate assessment difficult. To address this issue, we propose a novel Uncertainty-aware Reward Model (URM) that introduces a robust uncertainty estimation for the quality of paired responses based on Bayesian approximation. Trained with preference datasets, our uncertainty-enabled proxy not only scores rewards for responses but also evaluates their inherent uncertainty. Empirical results demonstrate significant benefits of incorporating the proposed proxy into language model training. Our method boosts the instruction following capability of language models by refining data curation for training and improving policy optimization objectives, thereby surpassing existing methods by a large margin on benchmarks such as Vicuna and MT-bench. These findings highlight that our proposed approach substantially advances language model training and paves a new way of harnessing uncertainty within language models. |
| title | Improving Instruction Following in Language Models through Proxy-Based Uncertainty Estimation |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2405.06424 |