Image de couverture de livre

Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Su, Dan, Kong, Kezhi, Lin, Ying, Jennings, Joseph, Norick, Brandon, Kliegl, Markus, Patwary, Mostofa, Shoeybi, Mohammad, Catanzaro, Bryan
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Nemotron-CC-Math: A 133 Billion-Token-Scale High Quality Math Pretraining Dataset
par: Mahabadi, Rabeeh Karimi, et autres
Publié: (2025)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
par: Feng, Steven, et autres
Publié: (2024)

Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models
par: Parmar, Jupinder, et autres
Publié: (2024)

Data, Data Everywhere: A Guide for Pretraining Dataset Construction
par: Parmar, Jupinder, et autres
Publié: (2024)

Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data
par: Akter, Syeda Nahida, et autres
Publié: (2025)

Nemotron-CrossThink: Scaling Self-Learning beyond Math Reasoning
par: Akter, Syeda Nahida, et autres
Publié: (2025)

RLP: Reinforcement as a Pretraining Objective
par: Hatamizadeh, Ali, et autres
Publié: (2025)

MIND: Math Informed syNthetic Dialogues for Pretraining LLMs
par: Akter, Syeda Nahida, et autres
Publié: (2024)

Nemotron Elastic: Towards Efficient Many-in-One Reasoning LLMs
par: Taghibakhshi, Ali, et autres
Publié: (2025)

Nemotron-4 15B Technical Report
par: Parmar, Jupinder, et autres
Publié: (2024)

Nemotron-CLIMB: CLustering-based Iterative Data Mixture Bootstrapping for Language Model Pre-training
par: Diao, Shizhe, et autres
Publié: (2025)

FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Data
par: Feng, Tao, et autres
Publié: (2025)

AceReason-Nemotron 1.1: Advancing Math and Code Reasoning through SFT and RL Synergy
par: Liu, Zihan, et autres
Publié: (2025)

AceReason-Nemotron: Advancing Math and Code Reasoning through Reinforcement Learning
par: Chen, Yang, et autres
Publié: (2025)

Compact Language Models via Pruning and Knowledge Distillation
par: Muralidharan, Saurav, et autres
Publié: (2024)

Retro-Search: Exploring Untaken Paths for Deeper and Efficient Reasoning
par: Lu, Ximing, et autres
Publié: (2025)

Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Models
par: Wang, Boxin, et autres
Publié: (2025)

InstructRetro: Instruction Tuning post Retrieval-Augmented Pretraining
par: Wang, Boxin, et autres
Publié: (2023)

AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
par: Liu, Zihan, et autres
Publié: (2024)

GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
par: Kargaran, Amir Hossein, et autres
Publié: (2024)

CC-GPX: Extracting High-Quality Annotated Geospatial Data from Common Crawl
par: Ilyankou, Ilya, et autres
Publié: (2024)

ChatQA 2: Bridging the Gap to Proprietary LLMs in Long Context and RAG Capabilities
par: Xu, Peng, et autres
Publié: (2024)

Prismatic Synthesis: Gradient-based Data Diversification Boosts Generalization in LLM Reasoning
par: Jung, Jaehun, et autres
Publié: (2025)

Nemotron-Cascade 2: Post-Training LLMs with Cascade RL and Multi-Domain On-Policy Distillation
par: Yang, Zhuolin, et autres
Publié: (2026)

Llama-Nemotron: Efficient Reasoning Models
par: Bercovich, Akhiad, et autres
Publié: (2025)

On Data Engineering for Scaling LLM Terminal Capabilities
par: Pi, Renjie, et autres
Publié: (2026)

From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Models
par: Xu, Chejian, et autres
Publié: (2025)

Nemotron-4 340B Technical Report
par: Nvidia, et autres
Publié: (2024)

MM-Embed: Universal Multimodal Retrieval with Multimodal LLMs
par: Lin, Sheng-Chieh, et autres
Publié: (2024)

Nemotron-Research-Tool-N1: Exploring Tool-Using Language Models with Reinforced Reasoning
par: Zhang, Shaokun, et autres
Publié: (2025)

NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models
par: Lee, Chankyu, et autres
Publié: (2024)

ChatQA: Surpassing GPT-4 on Conversational QA and RAG
par: Liu, Zihan, et autres
Publié: (2024)

RAVEN: In-Context Learning with Retrieval-Augmented Encoder-Decoder Language Models
par: Huang, Jie, et autres
Publié: (2023)

Quantifying Geospatial in the Common Crawl Corpus
par: Ilyankou, Ilya, et autres
Publié: (2024)

An Empirical Study of Mamba-based Language Models
par: Waleffe, Roger, et autres
Publié: (2024)

UnifiedCrawl: Aggregated Common Crawl for Affordable Adaptation of LLMs on Low-Resource Languages
par: Tessema, Bethel Melesse, et autres
Publié: (2024)

Retrieval meets Long Context Large Language Models
par: Xu, Peng, et autres
Publié: (2023)

Cybersecurity Data Extraction from Common Crawl
par: Mahara, Ashim
Publié: (2025)

NVIDIA Nemotron Parse 1.1
par: Chumachenko, Kateryna, et autres
Publié: (2025)

RankRAG: Unifying Context Ranking with Retrieval-Augmented Generation in LLMs
par: Yu, Yue, et autres
Publié: (2024)