Cover Image

Harnessing Diversity for Important Data Selection in Pretraining Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Chi, Zhong, Huaping, Zhang, Kuan, Chai, Chengliang, Wang, Rui, Zhuang, Xinlin, Bai, Tianyi, Qiu, Jiantao, Cao, Lei, Fan, Ju, Yuan, Ye, Wang, Guoren, He, Conghui
Format:	Preprint
Published:	2024
Subjects:	Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration
by: Bai, Tianyi, et al.
Published: (2024)

Not All Documents Are What You Need for Extracting Instruction Tuning Data
by: Zhang, Chi, et al.
Published: (2025)

Meta-rater: A Multi-dimensional Data Selection Method for Pre-training Language Models
by: Zhuang, Xinlin, et al.
Published: (2025)

Handling Label Noise via Instance-Level Difficulty Modeling and Dynamic Optimization
by: Zhang, Kuan, et al.
Published: (2025)

Topic Over Source: The Key to Effective Data Mixing for Language Models Pre-training
by: Peng, Jiahui, et al.
Published: (2025)

VADE: Variance-Aware Dynamic Sampling via Online Sample-Level Difficulty Estimation for Multimodal RL
by: Hu, Zengjie, et al.
Published: (2025)

QUEST: Query Optimization in Unstructured Document Analysis
by: Sun, Zhaoze, et al.
Published: (2025)

AgenticOCR: Parsing Only What You Need for Efficient Retrieval-Augmented Generation
by: Wang, Zhengren, et al.
Published: (2026)

Hallucination at a Glance: Controlled Visual Edits and Fine-Grained Multimodal Learning
by: Bai, Tianyi, et al.
Published: (2025)

Multi-Step Visual Reasoning with Visual Tokens Scaling and Verification
by: Bai, Tianyi, et al.
Published: (2025)

QuaDMix: Quality-Diversity Balanced Data Selection for Efficient LLM Pretraining
by: Liu, Fengze, et al.
Published: (2025)

Stop Looking for Important Tokens in Multimodal Language Models: Duplication Matters More
by: Wen, Zichen, et al.
Published: (2025)

SG-BEV: Satellite-Guided BEV Fusion for Cross-View Semantic Segmentation
by: Ye, Junyan, et al.
Published: (2024)

Cross-view image geo-localization with Panorama-BEV Co-Retrieval Network
by: Ye, Junyan, et al.
Published: (2024)

PACE: Poisoning Attacks on Learned Cardinality Estimation
by: Zhang, Jintao, et al.
Published: (2024)

AutoCE: An Accurate and Efficient Model Advisor for Learned Cardinality Estimation
by: Zhang, Jintao, et al.
Published: (2024)

Influence Maximization in Hypergraphs by Stratified Sampling for Efficient Generation of Reverse Reachable Sets
by: Zhang, Lingling, et al.
Published: (2024)

Unstructured Data Analysis using LLMs: A Comprehensive Benchmark
by: Deng, Qiyan, et al.
Published: (2025)

KeyVideoLLM: Towards Large-scale Video Keyframe Selection
by: Liang, Hao, et al.
Published: (2024)

StreamTGN: A GPU-Efficient Serving System for Streaming Temporal Graph Neural Networks
by: Zhang, Lingling, et al.
Published: (2026)

CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis
by: Li, Weijia, et al.
Published: (2024)

Graph-Based Feature Augmentation for Predictive Tasks on Relational Datasets
by: Qiao, Lianpeng, et al.
Published: (2025)

An Empirical Study on Influence-Based Pretraining Data Selection for Code Large Language Models
by: Xing, Chengli, et al.
Published: (2026)

DIffSteISR: Harnessing Diffusion Prior for Superior Real-world Stereo Image Super-Resolution
by: Zhou, Yuanbo, et al.
Published: (2024)

Not All Instances Are Equally Valuable: Towards Influence-Weighted Dataset Distillation
by: Deng, Qiyan, et al.
Published: (2025)

P‐122: Dependence of Aerial image quality on Display parameters in aerial Display system
by: Xinlin Ye, et al.
Published: (2024)

Multivariate Time Series Cleaning under Speed Constraints
by: Zhang, Aoqian, et al.
Published: (2024)

Matryoshka: Optimization of Dynamic Diverse Quantum Chemistry Systems via Elastic Parallelism Transformation
by: Wang, Tuowei, et al.
Published: (2024)

Multiomics Analyses Demonstrate the Attenuation of Metabolic Cardiac Disorders Associated With Type 2 Diabetes by Stachydrine in Relation With the Transition of Gastrointestinal Microbiota
by: Chaoxing Yang, et al.
Published: (2025)

An Aggregation‐Induced Emission Active Peptide‐Based Fluorescent Probe for Highly Selective and Sensitive Detection of Hg(II) Ions and Its Multifield Applications
by: Shiyi Xiong, et al.
Published: (2025)

Diverse polymorphs and phase transitions in van der Waals In$_2$Se$_3$
by: Liu, Mingfeng, et al.
Published: (2025)

Class-Imbalanced-Aware Adaptive Dataset Distillation for Scalable Pretrained Model on Credit Scoring
by: Li, Xia, et al.
Published: (2025)

BLISS: A Lightweight Bilevel Influence Scoring Method for Data Selection in Language Model Pretraining
by: Hao, Jie, et al.
Published: (2025)

The Ad‐Micellar Preparation of Para‐Aramid Nanofiber/Polyacrylate Nanocomposite
by: Mengyu Zhang, et al.
Published: (2025)

VIGC: Visual Instruction Generation and Correction
by: Wang, Bin, et al.
Published: (2023)

IKnow: Instruction-Knowledge-Aware Continual Pretraining for Effective Domain Adaptation
by: Zhang, Tianyi, et al.
Published: (2025)

Macformer: Transformer with Random Maclaurin Feature Attention
by: Guo, Yuhan, et al.
Published: (2024)

EvoGymCM: Harnessing Continuous Material Stiffness for Soft Robot Co-Design
by: Shen, Le, et al.
Published: (2026)

Scalable $k$-clique Densest Subgraph Search
by: Ye, Xiaowei, et al.
Published: (2024)

Harnessing Inherent Noises for Privacy Preservation in Quantum Machine Learning
by: Ju, Keyi, et al.
Published: (2023)