Beyond Line-Level Filtering for the Pretraining Corpora of LLMs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915582419402752 |
|---|---|
| author | Park, Chanwoo Park, Suyoung Ahn, Yelim Kim, Jongmin Park, Jongyeon Lee, Jaejin |
| author_facet | Park, Chanwoo Park, Suyoung Ahn, Yelim Kim, Jongmin Park, Jongyeon Lee, Jaejin |
| contents | While traditional line-level filtering techniques, such as line-level deduplication and trailing-punctuation filters, are commonly used, these basic methods can sometimes discard valuable content, negatively affecting downstream performance. In this paper, we introduce two methods-pattern-aware line-level deduplication (PLD) and pattern-aware trailing punctuation filtering (PTF)-by enhancing the conventional filtering techniques. Our approach not only considers line-level signals but also takes into account their sequential distribution across documents, enabling us to retain structurally important content that might otherwise be removed. We evaluate these proposed methods by training small language models (1 B parameters) in both English and Korean. The results demonstrate that our methods consistently improve performance on multiple-choice benchmarks and significantly enhance generative question-answering accuracy on both SQuAD v1 and KorQuAD v1. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24139 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Line-Level Filtering for the Pretraining Corpora of LLMs Park, Chanwoo Park, Suyoung Ahn, Yelim Kim, Jongmin Park, Jongyeon Lee, Jaejin Computation and Language Artificial Intelligence While traditional line-level filtering techniques, such as line-level deduplication and trailing-punctuation filters, are commonly used, these basic methods can sometimes discard valuable content, negatively affecting downstream performance. In this paper, we introduce two methods-pattern-aware line-level deduplication (PLD) and pattern-aware trailing punctuation filtering (PTF)-by enhancing the conventional filtering techniques. Our approach not only considers line-level signals but also takes into account their sequential distribution across documents, enabling us to retain structurally important content that might otherwise be removed. We evaluate these proposed methods by training small language models (1 B parameters) in both English and Korean. The results demonstrate that our methods consistently improve performance on multiple-choice benchmarks and significantly enhance generative question-answering accuracy on both SQuAD v1 and KorQuAD v1. |
| title | Beyond Line-Level Filtering for the Pretraining Corpora of LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.24139 |