Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.16397 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918218627547136 |
|---|---|
| author | Ma, Ren Qiu, Jiantao Xu, Chao Chu, Pei Liu, Kaiwen Ren, Pengli Qu, Yuan Peng, Jiahui Hou, Linfeng Liu, Mengjie Lu, Lindong Ning, Wenchang Yu, Jia Min, Rui Shi, Jin Chen, Haojiong Zhang, Peng Zhang, Wenjian Jiang, Qian Hu, Zengjie Yang, Guoqiang Li, Zhenxiang Shang, Fukai Ma, Runyuan Su, Chenlin Tu, Zhongying Zhang, Wentao Lin, Dahua He, Conghui |
| author_facet | Ma, Ren Qiu, Jiantao Xu, Chao Chu, Pei Liu, Kaiwen Ren, Pengli Qu, Yuan Peng, Jiahui Hou, Linfeng Liu, Mengjie Lu, Lindong Ning, Wenchang Yu, Jia Min, Rui Shi, Jin Chen, Haojiong Zhang, Peng Zhang, Wenjian Jiang, Qian Hu, Zengjie Yang, Guoqiang Li, Zhenxiang Shang, Fukai Ma, Runyuan Su, Chenlin Tu, Zhongying Zhang, Wentao Lin, Dahua He, Conghui |
| contents | While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_16397 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser Ma, Ren Qiu, Jiantao Xu, Chao Chu, Pei Liu, Kaiwen Ren, Pengli Qu, Yuan Peng, Jiahui Hou, Linfeng Liu, Mengjie Lu, Lindong Ning, Wenchang Yu, Jia Min, Rui Shi, Jin Chen, Haojiong Zhang, Peng Zhang, Wenjian Jiang, Qian Hu, Zengjie Yang, Guoqiang Li, Zhenxiang Shang, Fukai Ma, Runyuan Su, Chenlin Tu, Zhongying Zhang, Wentao Lin, Dahua He, Conghui Computation and Language While web data quality is crucial for large language models, most curation efforts focus on filtering and deduplication,treating HTML-to-text extraction as a fixed pre-processing step. Existing web corpora rely on heuristic-based extractors like Trafilatura, which struggle to preserve document structure and frequently corrupt structured elements such as formulas, codes, and tables. We hypothesize that improving extraction quality can be as impactful as aggressive filtering strategies for downstream performance. We introduce MinerU-HTML, a novel extraction pipeline that reformulates content extraction as a sequence labeling problem solved by a 0.6B-parameter language model. Unlike text-density heuristics, MinerU-HTML leverages semantic understanding and employs a two-stage formatting pipeline that explicitly categorizes semantic elements before converting to Markdown. Crucially, its model-based approach is inherently scalable, whereas heuristic methods offer limited improvement pathways. On MainWebBench, our benchmark of 7,887 annotated web pages, MinerU-HTML achieves 81.8\% ROUGE-N F1 compared to Trafilatura's 63.6\%, with exceptional structured element preservation (90.9\% for code blocks, 94.0\% for formulas). Using MinerU-HTML, we construct AICC (AI-ready Common Crawl), a 7.3-trillion token multilingual corpus from two Common Crawl snapshots. In controlled pretraining experiments where AICC and Trafilatura-extracted TfCC undergo identical filtering, models trained on AICC (62B tokens) achieve 50.8\% average accuracy across 13 benchmarks, outperforming TfCC by 1.08pp-providing direct evidence that extraction quality significantly impacts model capabilities. AICC also surpasses RefinedWeb and FineWeb on key benchmarks. We publicly release MainWebBench, MinerU-HTML, and AICC, demonstrating that HTML extraction is a critical, often underestimated component of web corpus construction. |
| title | AICC: Parse HTML Finer, Make Models Better -- A 7.3T AI-Ready Corpus Built by a Model-Based HTML Parser |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.16397 |