Imagen de Portada

Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Peng, Jiahui, Zhuang, Xinlin, Qiu, Jiantao, Ma, Ren, Yu, Jing, Zhu, He, He, Conghui
Formato:	Preprint
Publicado:	2025
Materias:	Computation and Language Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models
por: Zhuang, Xinlin, et al.
Publicado: (2025)

Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration
por: Bai, Tianyi, et al.
Publicado: (2024)

RegMix: Data Mixture as Regression for Language Model Pre-training
por: Liu, Qian, et al.
Publicado: (2024)

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training
por: Zhu, Tong, et al.
Publicado: (2024)

Efficient Data Learning for Open Information Extraction with Pre-trained Language Models
por: Fan, Zhiyuan, et al.
Publicado: (2023)

Evaluating Discourse Cohesion in Pre-trained Language Models
por: He, Jie, et al.
Publicado: (2025)

Generative Pre-trained Speech Language Model with Efficient Hierarchical Transformer
por: Zhu, Yongxin, et al.
Publicado: (2024)

Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models
por: Zheng, Junhao, et al.
Publicado: (2023)

FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models
por: Li, Wei, et al.
Publicado: (2024)

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
por: Yu, Hao, et al.
Publicado: (2026)

Spoken Language Identification with Pre-trained Models and Margin Loss
por: Fang, Zhihua, et al.
Publicado: (2026)

TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training
por: Jiang, Chaoya, et al.
Publicado: (2023)

Pre-trained Language Models and Few-shot Learning for Medical Entity Extraction
por: Wang, Xiaokai, et al.
Publicado: (2025)

Metadata Conditioning Accelerates Language Model Pre-training
por: Gao, Tianyu, et al.
Publicado: (2025)

Large Language Models Meet Symbolic Provers for Logical Reasoning Evaluation
por: Qi, Chengwen, et al.
Publicado: (2025)

Probing Language Models for Pre-training Data Detection
por: Liu, Zhenhua, et al.
Publicado: (2024)

Parallel Structures in Pre-training Data Yield In-Context Learning
por: Chen, Yanda, et al.
Publicado: (2024)

DataMan: Data Manager for Pre-training Large Language Models
por: Peng, Ru, et al.
Publicado: (2025)

Dripper: Token-Efficient Main HTML Extraction with a Lightweight LM
por: Liu, Mengjie, et al.
Publicado: (2025)

Model Merging in Pre-training of Large Language Models
por: Li, Yunshui, et al.
Publicado: (2025)

DSDL: Data Set Description Language for Bridging Modalities and Tasks in AI Data
por: Wang, Bin, et al.
Publicado: (2024)

Towards Label-Only Membership Inference Attack against Pre-trained Large Language Models
por: He, Yu, et al.
Publicado: (2025)

Enhancing Question Answering on Charts Through Effective Pre-training Tasks
por: Gupta, Ashim, et al.
Publicado: (2024)

Cross-layer Attention Sharing for Pre-trained Large Language Models
por: Mu, Yongyu, et al.
Publicado: (2024)

On Leveraging Encoder-only Pre-trained Language Models for Effective Keyphrase Generation
por: Wu, Di, et al.
Publicado: (2024)

WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages
por: Yu, Jia, et al.
Publicado: (2025)

Towards Effective and Efficient Continual Pre-training of Large Language Models
por: Chen, Jie, et al.
Publicado: (2024)

SoftDedup: an Efficient Data Reweighting Method for Speeding Up Language Model Pre-training
por: He, Nan, et al.
Publicado: (2024)

GeoX: Geometric Problem Solving Through Unified Formalized Vision-Language Pre-training
por: Xia, Renqiu, et al.
Publicado: (2024)

SampleMix: A Sample-wise Pre-training Data Mixing Strategey by Coordinating Data Quality and Diversity
por: Xi, Xiangyu, et al.
Publicado: (2025)

TRELM: Towards Robust and Efficient Pre-training for Knowledge-Enhanced Language Models
por: Yan, Junbing, et al.
Publicado: (2024)

DEPT: Decoupled Embeddings for Pre-training Language Models
por: Iacob, Alex, et al.
Publicado: (2024)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
por: Yang, Kailai, et al.
Publicado: (2025)

Investigating Data Contamination for Pre-training Language Models
por: Jiang, Minhao, et al.
Publicado: (2024)

Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation
por: Zhu, Hongyin, et al.
Publicado: (2021)

From Curated Data to Scalable Models: Continual Pre-training of Dense and MoE Large Language Models for Tibetan
por: Yang, Lei, et al.
Publicado: (2025)

Evolution of Concepts in Language Model Pre-Training
por: Ge, Xuyang, et al.
Publicado: (2025)

Masked Structural Growth for 2x Faster Language Model Pre-training
por: Yao, Yiqun, et al.
Publicado: (2023)

Harnessing Diversity for Important Data Selection in Pretraining Large Language Models
por: Zhang, Chi, et al.
Publicado: (2024)

Neural Topic Modeling with Large Language Models in the Loop
por: Yang, Xiaohao, et al.
Publicado: (2024)