Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.10795 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913431450288128 |
|---|---|
| author | Zhu, Wenhao Liu, Sizhe Huang, Shujian She, Shuaijie Wendler, Chris Chen, Jiajun |
| author_facet | Zhu, Wenhao Liu, Sizhe Huang, Shujian She, Shuaijie Wendler, Chris Chen, Jiajun |
| contents | Decoding by contrasting layers (DoLa), is designed to improve the generation quality of large language models (LLMs) by contrasting the prediction probabilities between an early exit output (amateur logits) and the final output (expert logits). However, we find that this approach does not work well on non-English tasks. Inspired by previous interpretability work on language transition during the model's forward pass, we discover that this issue arises from a language mismatch between early exit output and final output. In this work, we propose an improved contrastive decoding algorithm that is effective for diverse languages beyond English. To obtain more helpful amateur logits, we devise two strategies to skip a set of bottom, language-agnostic layers based on our preliminary analysis. Experimental results on multilingual reasoning benchmarks demonstrate that our proposed method outperforms previous contrastive decoding baselines and substantially improves LLM's chain-of-thought reasoning accuracy across 11 languages. The project will be available at: https://github.com/NJUNLP/SkipLayerCD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_10795 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping Zhu, Wenhao Liu, Sizhe Huang, Shujian She, Shuaijie Wendler, Chris Chen, Jiajun Computation and Language Decoding by contrasting layers (DoLa), is designed to improve the generation quality of large language models (LLMs) by contrasting the prediction probabilities between an early exit output (amateur logits) and the final output (expert logits). However, we find that this approach does not work well on non-English tasks. Inspired by previous interpretability work on language transition during the model's forward pass, we discover that this issue arises from a language mismatch between early exit output and final output. In this work, we propose an improved contrastive decoding algorithm that is effective for diverse languages beyond English. To obtain more helpful amateur logits, we devise two strategies to skip a set of bottom, language-agnostic layers based on our preliminary analysis. Experimental results on multilingual reasoning benchmarks demonstrate that our proposed method outperforms previous contrastive decoding baselines and substantially improves LLM's chain-of-thought reasoning accuracy across 11 languages. The project will be available at: https://github.com/NJUNLP/SkipLayerCD. |
| title | Multilingual Contrastive Decoding via Language-Agnostic Layers Skipping |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.10795 |