Copertina

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Behnia, Tina, Deora, Puneesh, Thrampoulidis, Christos
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly
di: Deora, Puneesh, et al.
Pubblicazione: (2025)

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
di: Vasudeva, Bhavya, et al.
Pubblicazione: (2026)

Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations
di: Zhao, Yize, et al.
Pubblicazione: (2024)

Implicit Bias and Fast Convergence Rates for Self-attention
di: Vasudeva, Bhavya, et al.
Pubblicazione: (2024)

On the Optimization and Generalization of Multi-head Attention
di: Deora, Puneesh, et al.
Pubblicazione: (2023)

Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features
di: Behnia, Tina, et al.
Pubblicazione: (2024)

How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
di: Vasudeva, Bhavya, et al.
Pubblicazione: (2025)

Implicit Optimization Bias of Next-Token Prediction in Linear Models
di: Thrampoulidis, Christos
Pubblicazione: (2024)

Unsupervised Pretraining for Fact Verification by Language Model Distillation
di: Bazaga, Adrián, et al.
Pubblicazione: (2023)

Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models
di: Deng, Wenlong, et al.
Pubblicazione: (2026)

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
di: Feng, Jiahai, et al.
Pubblicazione: (2024)

DARE the Extreme: Revisiting Delta-Parameter Pruning For Fine-Tuned Models
di: Deng, Wenlong, et al.
Pubblicazione: (2024)

On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
di: Deng, Wenlong, et al.
Pubblicazione: (2025)

The Heuristic Core: Understanding Subnetwork Generalization in Pretrained Language Models
di: Bhaskar, Adithya, et al.
Pubblicazione: (2024)

Transformers as Support Vector Machines
di: Tarzanagh, Davoud Ataee, et al.
Pubblicazione: (2023)

Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learning
di: Deng, Wenlong, et al.
Pubblicazione: (2025)

Leveraging Online Olympiad-Level Math Problems for LLMs Training and Contamination-Resistant Evaluation
di: Mahdavi, Sadegh, et al.
Pubblicazione: (2025)

On Large Language Models' Hallucination with Regard to Known Facts
di: Jiang, Che, et al.
Pubblicazione: (2024)

Modeling Caption Diversity in Contrastive Vision-Language Pretraining
di: Lavoie, Samuel, et al.
Pubblicazione: (2024)

LLM-Assisted Content Conditional Debiasing for Fair Text Embedding
di: Deng, Wenlong, et al.
Pubblicazione: (2024)

Factual Consistency of Multilingual Pretrained Language Models
di: Fierro, Constanza, et al.
Pubblicazione: (2022)

Jointly Reinforcing Diversity and Quality in Language Model Generations
di: Li, Tianjian, et al.
Pubblicazione: (2025)

What Happens When Small Is Made Smaller? Exploring the Impact of Compression on Small Data Pretrained Language Models
di: Awobade, Busayo, et al.
Pubblicazione: (2024)

Unlocking the Potential of Prompt-Tuning in Bridging Generalized and Personalized Federated Learning
di: Deng, Wenlong, et al.
Pubblicazione: (2023)

Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks
di: Fauber, Ben
Pubblicazione: (2024)

From Data to Knowledge: Evaluating How Efficiently Language Models Learn Facts
di: Christoph, Daniel, et al.
Pubblicazione: (2025)

On the Generalizability of "Competition of Mechanisms: Tracing How Language Models Handle Facts and Counterfactuals"
di: Dotsinski, Asen, et al.
Pubblicazione: (2025)

ESLM: Risk-Averse Selective Language Modeling for Efficient Pretraining
di: Bal, Melis Ilayda, et al.
Pubblicazione: (2025)

Analyzing Similarity Metrics for Data Selection for Language Model Pretraining
di: Sam, Dylan, et al.
Pubblicazione: (2025)

Procedural Pretraining: Warming Up Language Models with Abstract Data
di: Jiang, Liangze, et al.
Pubblicazione: (2026)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
di: Ruis, Laura, et al.
Pubblicazione: (2024)

RePro: Training Language Models to Faithfully Recycle the Web for Pretraining
di: Yu, Zichun, et al.
Pubblicazione: (2025)

Language Models Improve When Pretraining Data Matches Target Tasks
di: Mizrahi, David, et al.
Pubblicazione: (2025)

Temporal Entailment Pretraining for Clinical Language Models over EHR Data
di: Tanaka, Tatsunori, et al.
Pubblicazione: (2025)

Data Mixing for Large Language Models Pretraining: A Survey and Outlook
di: Chen, Zhuo, et al.
Pubblicazione: (2026)

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling
di: Grangier, David, et al.
Pubblicazione: (2024)

AF Adapter: Continual Pretraining for Building Chinese Biomedical Language Model
di: Yan, Yongyu, et al.
Pubblicazione: (2022)

How Does Code Pretraining Affect Language Model Task Performance?
di: Petty, Jackson, et al.
Pubblicazione: (2024)

Are Large Language Models Table-based Fact-Checkers?
di: Zhang, Hanwen, et al.
Pubblicazione: (2024)

Beyond Quality: Unlocking Diversity in Ad Headline Generation with Large Language Models
di: Wang, Chang, et al.
Pubblicazione: (2025)