Image de couverture de livre

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Sam, Dylan, Chakrabarti, Ayan, Rostamizadeh, Afshin, Ramalingam, Srikumar, Citovsky, Gui, Kumar, Sanjiv
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
par: Ye, Ke, et autres
Publié: (2024)

A Little Help Goes a Long Way: Efficient LLM Training by Leveraging Small LMs
par: Rawat, Ankit Singh, et autres
Publié: (2024)

DistillSpec: Improving Speculative Decoding via Knowledge Distillation
par: Zhou, Yongchao, et autres
Publié: (2023)

Rethinking FID: Towards a Better Evaluation Metric for Image Generation
par: Jayasumana, Sadeep, et autres
Publié: (2023)

SoftSRV: Learn to Generate Targeted Synthetic Data
par: DeSalvo, Giulia, et autres
Publié: (2024)

Language Models Improve When Pretraining Data Matches Target Tasks
par: Mizrahi, David, et autres
Publié: (2025)

GIST: Greedy Independent Set Thresholding for Max-Min Diversification with Submodular Utility
par: Fahrbach, Matthew, et autres
Publié: (2024)

Efficient Stagewise Pretraining via Progressive Subnetworks
par: Panigrahi, Abhishek, et autres
Publié: (2024)

Named Entity Recognition for Payment Data Using NLP
par: Nayak, Srikumar
Publié: (2026)

Latent Performance Profiling of Large Language Models
par: Chakraborty, Tanmoy, et autres
Publié: (2026)

MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
par: Yu, Zichun, et autres
Publié: (2024)

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
par: Messmer, Bettina, et autres
Publié: (2025)

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
par: Bal, Melis Ilayda, et autres
Publié: (2025)

Revisiting Multilingual Data Mixtures in Language Model Pretraining
par: Foroutan, Negar, et autres
Publié: (2025)

Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe
par: You, Chong, et autres
Publié: (2025)

Group-Level Data Selection for Efficient Pretraining
par: Yu, Zichun, et autres
Publié: (2025)

SOI Matters: Analyzing Multi-Setting Training Dynamics in Pretrained Language Models via Subsets of Interest
par: Vassef, Shayan, et autres
Publié: (2025)

Learning Translations: Emergent Communication Pretraining for Cooperative Language Acquisition
par: Cope, Dylan, et autres
Publié: (2024)

Towards Safer Pretraining: Analyzing and Filtering Harmful Content in Webscale datasets for Responsible LLMs
par: Mendu, Sai Krishna, et autres
Publié: (2025)

Procedural Pretraining: Warming Up Language Models with Abstract Data
par: Jiang, Liangze, et autres
Publié: (2026)

HQFS: Hybrid Quantum Classical Financial Security with VQC Forecasting, QUBO Annealing, and Audit-Ready Post-Quantum Signing
par: Nayak, Srikumar
Publié: (2026)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
par: Tanaka, Tatsunori, et autres
Publié: (2025)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
par: Chen, Zhuo, et autres
Publié: (2026)

Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
par: Xie, Wanyun, et autres
Publié: (2025)

MuRating: A High Quality Data Selecting Approach to Multilingual Large Language Model Pretraining
par: Chen, Zhixun, et autres
Publié: (2025)

Detecting Pretraining Data from Large Language Models
par: Shi, Weijia, et autres
Publié: (2023)

Promises and Pitfalls of Generative Masked Language Modeling: Theoretical Framework and Practical Guidelines
par: Li, Yuchen, et autres
Publié: (2024)

Neural Diversity Regularizes Hallucinations in Language Models
par: Chakrabarti, Kushal, et autres
Publié: (2025)

Algorithms for Learning Kernels Based on Centered Alignment
par: Cortes, Corinna, et autres
Publié: (2012)

Persona-aware Generative Model for Code-mixed Language
par: Sengupta, Ayan, et autres
Publié: (2023)

Mimetic Initialization Helps State Space Models Learn to Recall
par: Trockman, Asher, et autres
Publié: (2024)

Factual Consistency of Multilingual Pretrained Language Models
par: Fierro, Constanza, et autres
Publié: (2022)

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
par: Krajewski, Jakub, et autres
Publié: (2025)

RLShield: Practical Multi-Agent RL for Financial Cyber Defense with Attack-Surface MDPs and Real-Time Response Orchestration
par: Nayak, Srikumar
Publié: (2026)

A Survey on Data Selection for Language Models
par: Albalak, Alon, et autres
Publié: (2024)

PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models
par: Agiza, Ahmed, et autres
Publié: (2024)

Objective Metrics for Evaluating Large Language Models Using External Data Sources
par: Du, Haoze, et autres
Publié: (2025)

Language Model Cascades: Token-level uncertainty and beyond
par: Gupta, Neha, et autres
Publié: (2024)

Unsupervised Pretraining for Fact Verification by Language Model Distillation
par: Bazaga, Adrián, et autres
Publié: (2023)

Mordal: Automated Pretrained Model Selection for Vision Language Models
par: He, Shiqi, et autres
Publié: (2025)