Imagen de Portada

A Causal Language Modeling Detour Improves Encoder Continued Pretraining

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Touchent, Rian, de la Clergerie, Eric
Formato:	Preprint
Publicado:	2026
Materias:	Computation and Language Artificial Intelligence
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data
por: Touchent, Rian, et al.
Publicado: (2023)

Biomed-Enriched: A Biomedical Dataset Enriched with LLMs for Pretraining and Extracting Rare and Hidden Content
por: Touchent, Rian, et al.
Publicado: (2025)

Gaperon: A Peppered English-French Generative Language Model Suite
por: Godey, Nathan, et al.
Publicado: (2025)

CamemBERT 2.0: A Smarter French Language Model Aged to Perfection
por: Antoun, Wissam, et al.
Publicado: (2024)

On the Scaling Laws of Geographical Representation in Language Models
por: Godey, Nathan, et al.
Publicado: (2024)

Instruct-Tuning Pretrained Causal Language Models for Ancient Greek Papyrology and Epigraphy
por: Cullhed, Eric
Publicado: (2024)

PatentEval: Understanding Errors in Patent Generation
por: Zuo, You, et al.
Publicado: (2024)

Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression
por: Godey, Nathan, et al.
Publicado: (2025)

PonderLM: Pretraining Language Models to Ponder in Continuous Space
por: Zeng, Boyi, et al.
Publicado: (2025)

Code Pretraining Improves Entity Tracking Abilities of Language Models
por: Kim, Najoung, et al.
Publicado: (2024)

Emergent Abilities of Large Language Models under Continued Pretraining for Language Adaptation
por: Elhady, Ahmed, et al.
Publicado: (2025)

Renaissance: Investigating the Pretraining of Vision-Language Encoders
por: Fields, Clayton, et al.
Publicado: (2024)

Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages
por: Su, Zeli, et al.
Publicado: (2025)

The Tokenization Bottleneck: How Vocabulary Extension Improves Chemistry Representation Learning in Pretrained Language Models
por: Kalamkar, Prathamesh, et al.
Publicado: (2025)

Multilingual Pretraining for Pixel Language Models
por: Kesen, Ilker, et al.
Publicado: (2025)

Ayn: A Tiny yet Competitive Indian Legal Language Model Pretrained from Scratch
por: Niyogi, Mitodru, et al.
Publicado: (2024)

Language Models as Hierarchy Encoders
por: He, Yuan, et al.
Publicado: (2024)

On Linear Representations and Pretraining Data Frequency in Language Models
por: Merullo, Jack, et al.
Publicado: (2025)

Drop Dropout on Single-Epoch Language Model Pretraining
por: Liu, Houjun, et al.
Publicado: (2025)

Gender Encoding Patterns in Pretrained Language Model Representations
por: Zakizadeh, Mahdi, et al.
Publicado: (2025)

Rethinking the Role of Text Complexity in Language Model Pretraining
por: Velasco, Dan John, et al.
Publicado: (2025)

Has Your Pretrained Model Improved? A Multi-head Posterior Based Approach
por: Aboagye, Prince, et al.
Publicado: (2024)

Sparse Auto-Encoders and Holism about Large Language Models
por: Grindrod, Jumbly
Publicado: (2026)

ParaSpeechCLAP: A Dual-Encoder Speech-Text Model for Rich Stylistic Language-Audio Pretraining
por: Diwan, Anuj, et al.
Publicado: (2026)

Why do small language models underperform? Studying Language Model Saturation via the Softmax Bottleneck
por: Godey, Nathan, et al.
Publicado: (2024)

Injecting Structured Biomedical Knowledge into Language Models: Continual Pretraining vs. GraphRAG
por: Klila, Jaafer, et al.
Publicado: (2026)

NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models
por: Gong, Haisong, et al.
Publicado: (2026)

A Continued Pretrained LLM Approach for Automatic Medical Note Generation
por: Yuan, Dong, et al.
Publicado: (2024)

Foundation CAN LM: A Pretrained Language Model For Automotive CAN Data
por: Esashi, Akiharu, et al.
Publicado: (2026)

AraPoemBERT: A Pretrained Language Model for Arabic Poetry Analysis
por: Qarah, Faisal
Publicado: (2024)

Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
por: Cao, Jiaqi, et al.
Publicado: (2025)

SaudiBERT: A Large Language Model Pretrained on Saudi Dialect Corpora
por: Qarah, Faisal
Publicado: (2024)

BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
por: Boizard, Nicolas, et al.
Publicado: (2026)

ptt5-v2: A Closer Look at Continued Pretraining of T5 Models for the Portuguese Language
por: Piau, Marcos, et al.
Publicado: (2024)

CausalEval: Towards Better Causal Reasoning in Language Models
por: Yu, Longxuan, et al.
Publicado: (2024)

Improving Legal Judgement Prediction in Romanian with Long Text Encoders
por: Masala, Mihai, et al.
Publicado: (2024)

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
por: Yu, Huimu, et al.
Publicado: (2024)

Knowledge Graph Structure as Prompt: Improving Small Language Models Capabilities for Knowledge-based Causal Discovery
por: Susanti, Yuni, et al.
Publicado: (2024)

Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling
por: Egli, Eric, et al.
Publicado: (2025)

Exploring the Benefits of Domain-Pretraining of Generative Large Language Models for Chemistry
por: Acharya, Anurag, et al.
Publicado: (2024)