Cover Image

Should We Still Pretrain Encoders with Masked Language Modeling?

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gisserot-Boukhlef, Hippolyte, Boizard, Nicolas, Faysse, Manuel, Alves, Duarte M., Malherbe, Emmanuel, Martins, André F. T., Hudelot, Céline, Colombo, Pierre
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluation
by: Gisserot-Boukhlef, Hippolyte, et al.
Published: (2026)

Towards Trustworthy Reranking: A Simple yet Effective Abstention Mechanism
by: Gisserot-Boukhlef, Hippolyte, et al.
Published: (2024)

BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
by: Boizard, Nicolas, et al.
Published: (2026)

When Does Reasoning Matter? A Controlled Study of Reasoning's Contribution to Model Performance
by: Boizard, Nicolas, et al.
Published: (2025)

Is Preference Alignment Always the Best Option to Enhance LLM-Based Translation? An Empirical Analysis
by: Gisserot-Boukhlef, Hippolyte, et al.
Published: (2024)

EuroBERT: Scaling Multilingual Encoders for European Languages
by: Boizard, Nicolas, et al.
Published: (2025)

Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs
by: Boizard, Nicolas, et al.
Published: (2024)

Context is Gold to find the Gold Passage: Evaluating and Training Contextual Document Embeddings
by: Conti, Max, et al.
Published: (2025)

CroissantLLM: A Truly Bilingual French-English Language Model
by: Faysse, Manuel, et al.
Published: (2024)

ColPali: Efficient Document Retrieval with Vision Language Models
by: Faysse, Manuel, et al.
Published: (2024)

EuroLLM-22B: Technical Report
by: Ramos, Miguel Moura, et al.
Published: (2026)

EuroLLM-9B: Technical Report
by: Martins, Pedro Henrique, et al.
Published: (2025)

EuroLLM: Multilingual Language Models for Europe
by: Martins, Pedro Henrique, et al.
Published: (2024)

When Should We Introduce Safety Interventions During Pretraining?
by: Sam, Dylan, et al.
Published: (2026)

ConceptGuard: Neuro-Symbolic Safety Guardrails via Sparse Interpretable Jailbreak Concepts
by: Aswal, Darpan, et al.
Published: (2025)

ModernVBERT: Towards Smaller Visual Document Retrievers
by: Teiletche, Paul, et al.
Published: (2025)

ARC-Encoder: learning compressed text representations for large language models
by: Pilchen, Hippolyte, et al.
Published: (2025)

ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarios
by: Loison, António, et al.
Published: (2026)

Time Series Forecasting Through the Lens of Dynamics
by: Brachet, Alexis-Raja, et al.
Published: (2025)

Bond Graphs for multi-physics informed Neural Networks for multi-variate time series
by: Brachet, Alexis-Raja, et al.
Published: (2024)

Revisiting Anisotropy in Language Transformers: The Geometry of Learning Dynamics
by: Bernas, Raphael, et al.
Published: (2026)

ViDoRe Benchmark V2: Raising the Bar for Visual Retrieval
by: Macé, Quentin, et al.
Published: (2025)

Renaissance: Investigating the Pretraining of Vision-Language Encoders
by: Fields, Clayton, et al.
Published: (2024)

Combining On-Policy Optimization and Distillation for Long-Context Reasoning in Large Language Models
by: Ramos, Miguel Moura, et al.
Published: (2026)

3D Feature Prediction for Masked-AutoEncoder-Based Point Cloud Pretraining
by: Yan, Siming, et al.
Published: (2023)

Are We Still Missing an Item?
by: Magen, Roey
Published: (2024)

‘We Are Still Challenged to be Dissatisfied’
by: The Editors
Published: (2011)

A Complexity Map of Probabilistic Reasoning for Neurosymbolic Classification Techniques
by: Ledaguenel, Arthur, et al.
Published: (2024)

Improving Neural-based Classification with Logical Background Knowledge
by: Ledaguenel, Arthur, et al.
Published: (2024)

Neurosymbolic Conformal Classification
by: Ledaguenel, Arthur, et al.
Published: (2024)

Recommendation of data-free class-incremental learning algorithms by simulating future data
by: Feillet, Eva, et al.
Published: (2024)

Copyright Traps for Large Language Models
by: Meeus, Matthieu, et al.
Published: (2024)

Do we need rebalancing strategies? A theoretical and empirical study around SMOTE and its variants
by: Sakho, Abdoulaye, et al.
Published: (2024)

Discovering Interpretable Biological Concepts in Single-cell RNA-seq Foundation Models
by: Claye, Charlotte, et al.
Published: (2025)

How Should We Model the Probability of a Language?
by: Dent, Rasul, et al.
Published: (2026)

Building Solidarity With and Without Community Psychology
by: Nick Malherbe
Published: (2025)

Responsabilidad social de la empresa y fiscalidad
by: Jacques Malherbe
Published: (2013)

Data Supporting Figures for: Primordial Gravitational Wave Backgrounds from Phase Transitions with Next Generation Ground Based Detectors
by: Quelquejay, Hippolyte
Published: (2025)

Nonperforming loans in Sub-Saharan Africa : causal analysis and macroeconomic implications / Hippolyte Fofack
by: Fofack, Hippolyte
Published: (2005)

Dynamics of income inequality and welfare in Latvia in the late 1990s / Hippolyte Fofack, Celestin Monga
by: Fofack, Hippolyte
Published: (2004)