Image de couverture de livre

Embedding And Clustering Your Data Can Improve Contrastive Pretraining

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteur principal:	Merrick, Luke
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Arctic-Embed 2.0: Multilingual Retrieval Without Compromise
par: Yu, Puxuan, et autres
Publié: (2024)

Improving Pretraining Data Using Perplexity Correlations
par: Thrush, Tristan, et autres
Publié: (2024)

NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text
par: Kailas, Prajwal, et autres
Publié: (2024)

BeyondWeb: Lessons from Scaling Synthetic Data for Trillion-scale Pretraining
par: DatologyAI, et autres
Publié: (2025)

Language Models Improve When Pretraining Data Matches Target Tasks
par: Mizrahi, David, et autres
Publié: (2025)

Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretraining
par: Feng, Steven, et autres
Publié: (2024)

An Improved Deep Learning Model for Word Embeddings Based Clustering for Large Text Datasets
par: Sutrakar, Vijay Kumar, et autres
Publié: (2025)

How Learning Rate Decay Wastes Your Best Data in Curriculum-Based LLM Pretraining
par: Luo, Kairong, et autres
Publié: (2025)

Diffusion-Pretrained Dense and Contextual Embeddings
par: Eslami, Sedigheh, et autres
Publié: (2026)

Detecting Pretraining Data from Large Language Models
par: Shi, Weijia, et autres
Publié: (2023)

Efficient and Flexible Topic Modeling using Pretrained Embeddings and Bag of Sentences
par: Schneider, Johannes
Publié: (2023)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
par: Grangier, David, et autres
Publié: (2024)

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
par: Li, Ziyue, et autres
Publié: (2024)

Data-Centric Lessons To Improve Speech-Language Pretraining
par: Udandarao, Vishaal, et autres
Publié: (2025)

SDEC: Semantic Deep Embedded Clustering
par: Rahman, Mohammad Wali Ur, et autres
Publié: (2025)

Output Embedding Centering for Stable LLM Pretraining
par: Stollenwerk, Felix, et autres
Publié: (2026)

Group-Level Data Selection for Efficient Pretraining
par: Yu, Zichun, et autres
Publié: (2025)

Improving Clustering on Occupational Text Data through Dimensionality Reduction
par: García, Iago Xabier Vázquez, et autres
Publié: (2025)

TRACE: TRansformer-based Attribution using Contrastive Embeddings in LLMs
par: Wang, Cheng, et autres
Publié: (2024)

Self-Adaptive Reconstruction with Contrastive Learning for Unsupervised Sentence Embeddings
par: Liu, Junlong, et autres
Publié: (2024)

Can GRPO Help LLMs Transcend Their Pretraining Origin?
par: Ni, Kangqi, et autres
Publié: (2025)

DataDecide: How to Predict Best Pretraining Data with Small Experiments
par: Magnusson, Ian, et autres
Publié: (2025)

Improving Sentence Embeddings with Automatic Generation of Training Data Using Few-shot Examples
par: Sato, Soma, et autres
Publié: (2024)

Scaling Laws for Mixture Pretraining Under Data Constraints
par: Sedova, Anastasiia, et autres
Publié: (2026)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
par: Ma, Xuezhe, et autres
Publié: (2024)

Federated Learning for ICD Classification with Lightweight Models and Pretrained Embeddings
par: Xu, Binbin, et autres
Publié: (2025)

Pooling Attention: Evaluating Pretrained Transformer Embeddings for Deception Classification
par: Mamtani, Sumit, et autres
Publié: (2025)

Repetition Improves Language Model Embeddings
par: Springer, Jacob Mitchell, et autres
Publié: (2024)

MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models
par: Yu, Zichun, et autres
Publié: (2024)

Perturb Your Data: Paraphrase-Guided Training Data Watermarking
par: Shetty, Pranav, et autres
Publié: (2025)

Procedural Pretraining: Warming Up Language Models with Abstract Data
par: Jiang, Liangze, et autres
Publié: (2026)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
par: Bethune, Louis, et autres
Publié: (2025)

Enhancing Multilingual LLM Pretraining with Model-Based Data Selection
par: Messmer, Bettina, et autres
Publié: (2025)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
par: Sam, Dylan, et autres
Publié: (2025)

How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
par: Niklaus, Joel, et autres
Publié: (2026)

Improving Language Plasticity via Pretraining with Active Forgetting
par: Chen, Yihong, et autres
Publié: (2023)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
par: Chen, Zhuo, et autres
Publié: (2026)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
par: Tanaka, Tatsunori, et autres
Publié: (2025)

HU at SemEval-2024 Task 8A: Can Contrastive Learning Learn Embeddings to Detect Machine-Generated Text?
par: Dipta, Shubhashis Roy, et autres
Publié: (2024)

Luxical: High-Speed Lexical-Dense Text Embeddings
par: DatologyAI, et autres
Publié: (2025)