Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.06604 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918021900009472 |
|---|---|
| author | Ru, Jinghan Xie, Yuxin Zhuang, Xianwei Yin, Yuguo Guo, Zhihui Liu, Zhiming Ren, Qianli Zou, Yuexian |
| author_facet | Ru, Jinghan Xie, Yuxin Zhuang, Xianwei Yin, Yuguo Guo, Zhihui Liu, Zhiming Ren, Qianli Zou, Yuexian |
| contents | Web-scale pre-training datasets are the cornerstone of LLMs' success. However, text data curated from the Internet inevitably contains random noise caused by decoding errors or unregulated web content. In contrast to previous works that focus on low quality or synthetic data, our study \textbf{provides the first systematic investigation of such random noise through a cohesive ``What-Why-How'' framework.} Surprisingly, we observed that the resulting increase in the loss of next-token prediction (NTP) was significantly lower than the proportion of random noise even when the model was scaled up to 2.7B. We provide a theoretical justification for this phenomenon, which also elucidates the success of multilingual models and can be applied to multimodal models. On the other hand, experiments show that the model's performance in downstream tasks is not based solely on the NTP loss, which means that random noise may result in degraded downstream performance. To address the potential adverse effects, we introduce a novel plug-and-play Local Gradient Matching loss, which explicitly enhances the denoising capability of the downstream task head by aligning the gradient of normal and perturbed features without requiring knowledge of the model's parameters. Additional experiments on 8 language and 14 vision benchmarks further validate its effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_06604 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do we really have to filter out random noise in pre-training data for language models? Ru, Jinghan Xie, Yuxin Zhuang, Xianwei Yin, Yuguo Guo, Zhihui Liu, Zhiming Ren, Qianli Zou, Yuexian Computation and Language Web-scale pre-training datasets are the cornerstone of LLMs' success. However, text data curated from the Internet inevitably contains random noise caused by decoding errors or unregulated web content. In contrast to previous works that focus on low quality or synthetic data, our study \textbf{provides the first systematic investigation of such random noise through a cohesive ``What-Why-How'' framework.} Surprisingly, we observed that the resulting increase in the loss of next-token prediction (NTP) was significantly lower than the proportion of random noise even when the model was scaled up to 2.7B. We provide a theoretical justification for this phenomenon, which also elucidates the success of multilingual models and can be applied to multimodal models. On the other hand, experiments show that the model's performance in downstream tasks is not based solely on the NTP loss, which means that random noise may result in degraded downstream performance. To address the potential adverse effects, we introduce a novel plug-and-play Local Gradient Matching loss, which explicitly enhances the denoising capability of the downstream task head by aligning the gradient of normal and perturbed features without requiring knowledge of the model's parameters. Additional experiments on 8 language and 14 vision benchmarks further validate its effectiveness. |
| title | Do we really have to filter out random noise in pre-training data for language models? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.06604 |