Cover Image

Domain Pre-training Impact on Representations

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gonzalez-Gutierrez, Cesar, Quattoni, Ariadna
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

ZOGRASCOPE: A New Benchmark for Semantic Parsing over Property Graphs
by: Cazzaro, Francesco, et al.
Published: (2025)

Pre-training Language Model Incorporating Domain-specific Heterogeneous Knowledge into A Unified Representation
by: Zhu, Hongyin, et al.
Published: (2021)

PART: Pre-trained Authorship Representation Transformer
by: Huertas-Tato, Javier, et al.
Published: (2022)

To Code, or Not To Code? Exploring Impact of Code in Pre-training
by: Aryabumi, Viraat, et al.
Published: (2024)

Efficient Continual Pre-training for Building Domain Specific Large Language Models
by: Xie, Yong, et al.
Published: (2023)

Do Prompts Reshape Representations? An Empirical Study of Prompting Effects on Embeddings
by: Gonzalez-Gutierrez, Cesar, et al.
Published: (2025)

DocSum: Domain-Adaptive Pre-training for Document Abstractive Summarization
by: Chau, Phan Phuong Mai, et al.
Published: (2024)

Domain-Adapted Pre-trained Language Models for Implicit Information Extraction in Crash Narratives
by: Wang, Xixi, et al.
Published: (2025)

Velocitune: A Velocity-based Dynamic Domain Reweighting Method for Continual Pre-training
by: Luo, Zheheng, et al.
Published: (2024)

G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks
by: Wan, Zhongwei, et al.
Published: (2022)

Mecellem Models: Turkish Models Trained from Scratch and Continually Pre-trained for the Legal Domain
by: Uğur, Özgür, et al.
Published: (2026)

Superpixel Semantics Representation and Pre-training for Vision-Language Task
by: Zhang, Siyu, et al.
Published: (2023)

Construction of Domain-specified Japanese Large Language Model for Finance through Continual Pre-training
by: Hirano, Masanori, et al.
Published: (2024)

Pre-trained Language Models Learn Remarkably Accurate Representations of Numbers
by: Kadlčík, Marek, et al.
Published: (2025)

A Comparative Study of Pre-training and Self-training
by: Wang, Yiheng, et al.
Published: (2024)

On Predicting the Post-training Potential of Pre-trained LLMs
by: Li, Xiaoyuan, et al.
Published: (2026)

Understanding Data Temporality Impact on Large Language Models Pre-training
by: Pilchen, Hippolyte, et al.
Published: (2026)

REZE: Representation Regularization for Domain-adaptive Text Embedding Pre-finetuning
by: Lee, Seungmin, et al.
Published: (2026)

ColBERT-Zero: To Pre-train Or Not To Pre-train ColBERT models
by: Chaffin, Antoine, et al.
Published: (2026)

Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization
by: Zhang, Haode, et al.
Published: (2022)

IMO: Greedy Layer-Wise Sparse Representation Learning for Out-of-Distribution Text Classification with Pre-trained Models
by: Feng, Tao, et al.
Published: (2024)

Pre-training Cross-lingual Open Domain Question Answering with Large-scale Synthetic Supervision
by: Jiang, Fan, et al.
Published: (2024)

Can Continual Pre-training Bridge the Performance Gap between General-purpose and Specialized Language Models in the Medical Domain?
by: Doll, Niclas, et al.
Published: (2026)

AfriqueLLM: How Data Mixing and Model Architecture Impact Continued Pre-training for African Languages
by: Yu, Hao, et al.
Published: (2026)

Tackling Fake News in Bengali: Unraveling the Impact of Summarization vs. Augmentation on Pre-trained Language Models
by: Chowdhury, Arman Sakif, et al.
Published: (2023)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
by: Yang, Kailai, et al.
Published: (2025)

Enhancing EEG-to-Text Decoding through Transferable Representations from Pre-trained Contrastive EEG-Text Masked Autoencoder
by: Wang, Jiaqi, et al.
Published: (2024)

MS-HuBERT: Mitigating Pre-training and Inference Mismatch in Masked Language Modelling methods for learning Speech Representations
by: Yadav, Hemant, et al.
Published: (2024)

Scaling Agents via Continual Pre-training
by: Su, Liangcai, et al.
Published: (2025)

PhoGPT: Generative Pre-training for Vietnamese
by: Nguyen, Dat Quoc, et al.
Published: (2023)

UniWav: Towards Unified Pre-training for Speech Representation Learning and Generation
by: Liu, Alexander H., et al.
Published: (2025)

Thinking Augmented Pre-training
by: Wang, Liang, et al.
Published: (2025)

SuPreME: A Supervised Pre-training Framework for Multimodal ECG Representation Learning
by: Cai, Mingsheng, et al.
Published: (2025)

Enhancing Biomedical Multi-modal Representation Learning with Multi-scale Pre-training and Perturbed Report Discrimination
by: Zhong, Xinliu, et al.
Published: (2025)

Metadata Conditioning Accelerates Language Model Pre-training
by: Gao, Tianyu, et al.
Published: (2025)

Evaluating Discourse Cohesion in Pre-trained Language Models
by: He, Jie, et al.
Published: (2025)

Efficient Continual Pre-training by Mitigating the Stability Gap
by: Guo, Yiduo, et al.
Published: (2024)

New Intent Discovery with Pre-training and Contrastive Learning
by: Zhang, Yuwei, et al.
Published: (2022)

FaBERT: Pre-training BERT on Persian Blogs
by: Masumi, Mostafa, et al.
Published: (2024)

Effectiveness of Pre-training for Few-shot Intent Classification
by: Zhang, Haode, et al.
Published: (2021)