Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912920040898560 |
|---|---|
| author | Li, Jeffrey Gardner, Josh Kang, Doug Shi, Fangping Singh, Karanjeet Li, Chun-Liang Shandilya, Herumb Hall, David Tuzel, Oncel Liang, Percy Schmidt, Ludwig Ansari, Hadi Pour Faghri, Fartash |
| author_facet | Li, Jeffrey Gardner, Josh Kang, Doug Shi, Fangping Singh, Karanjeet Li, Chun-Liang Shandilya, Herumb Hall, David Tuzel, Oncel Liang, Percy Schmidt, Ludwig Ansari, Hadi Pour Faghri, Fartash |
| contents | One of the first pre-processing steps for constructing web-scale LLM pretraining datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71% while maintaining benchmark performance. We further show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance, with differences of up to 10 percentage points (p.p.) on WikiTQ and 3 p.p. on HumanEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_19548 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining Li, Jeffrey Gardner, Josh Kang, Doug Shi, Fangping Singh, Karanjeet Li, Chun-Liang Shandilya, Herumb Hall, David Tuzel, Oncel Liang, Percy Schmidt, Ludwig Ansari, Hadi Pour Faghri, Fartash Computation and Language Machine Learning One of the first pre-processing steps for constructing web-scale LLM pretraining datasets involves extracting text from HTML. Despite the immense diversity of web content, existing open-source datasets predominantly apply a single fixed extractor to all webpages. In this work, we investigate whether this practice leads to suboptimal coverage and utilization of Internet data. We first show that while different extractors may lead to similar model performance on standard language understanding tasks, the pages surviving a fixed filtering pipeline can differ substantially. This suggests a simple intervention: by taking a Union over different extractors, we can increase the token yield of DCLM-Baseline by up to 71% while maintaining benchmark performance. We further show that for structured content such as tables and code blocks, extractor choice can significantly impact downstream task performance, with differences of up to 10 percentage points (p.p.) on WikiTQ and 3 p.p. on HumanEval. |
| title | Beyond a Single Extractor: Re-thinking HTML-to-Text Extraction for LLM Pretraining |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2602.19548 |