Top-$nσ$: Not All Logits Are You Need
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910695525711872 |
|---|---|
| author | Tang, Chenxia Liu, Jianchun Xu, Hongli Huang, Liusheng |
| author_facet | Tang, Chenxia Liu, Jianchun Xu, Hongli Huang, Liusheng |
| contents | Large language models (LLMs) typically employ greedy decoding or low-temperature sampling for reasoning tasks, reflecting a perceived trade-off between diversity and accuracy. We challenge this convention by introducing top-$nσ$, a novel sampling method that operates directly on pre-softmax logits by leveraging a statistical threshold. Our key insight is that logits naturally separate into a Gaussian-distributed noisy region and a distinct informative region, enabling efficient token filtering without complex probability manipulations. Unlike existing methods (e.g., top-$p$, min-$p$) that inadvertently include more noise tokens at higher temperatures, top-$nσ$ maintains a stable sampling space regardless of temperature scaling. We also provide a theoretical analysis of top-$nσ$ to better understand its behavior. The extensive experimental results across four reasoning-focused datasets demonstrate that our method not only outperforms existing sampling approaches but also surpasses greedy decoding, while maintaining consistent performance even at high temperatures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_07641 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Top-$nσ$: Not All Logits Are You Need Tang, Chenxia Liu, Jianchun Xu, Hongli Huang, Liusheng Machine Learning Large language models (LLMs) typically employ greedy decoding or low-temperature sampling for reasoning tasks, reflecting a perceived trade-off between diversity and accuracy. We challenge this convention by introducing top-$nσ$, a novel sampling method that operates directly on pre-softmax logits by leveraging a statistical threshold. Our key insight is that logits naturally separate into a Gaussian-distributed noisy region and a distinct informative region, enabling efficient token filtering without complex probability manipulations. Unlike existing methods (e.g., top-$p$, min-$p$) that inadvertently include more noise tokens at higher temperatures, top-$nσ$ maintains a stable sampling space regardless of temperature scaling. We also provide a theoretical analysis of top-$nσ$ to better understand its behavior. The extensive experimental results across four reasoning-focused datasets demonstrate that our method not only outperforms existing sampling approaches but also surpasses greedy decoding, while maintaining consistent performance even at high temperatures. |
| title | Top-$nσ$: Not All Logits Are You Need |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.07641 |