BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918425601769472 |
|---|---|
| author | Boizard, Nicolas Deschamps-Berger, Théo Gisserot-Boukhlef, Hippolyte Hudelot, Céline Colombo, Pierre |
| author_facet | Boizard, Nicolas Deschamps-Berger, Théo Gisserot-Boukhlef, Hippolyte Hudelot, Céline Colombo, Pierre |
| contents | Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_02045 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs Boizard, Nicolas Deschamps-Berger, Théo Gisserot-Boukhlef, Hippolyte Hudelot, Céline Colombo, Pierre Computation and Language Artificial Intelligence Transforming causal generative language models into bidirectional encoders offers a powerful alternative to BERT-style architectures. However, current approaches remain limited: they lack consensus on optimal training objectives, suffer from catastrophic forgetting at scale, and fail to flexibly integrate the vast ecosystem of specialized generative models. In this work, through systematic ablations on the Gemma3 and Qwen3 families, we identify the key factors driving successful adaptation, highlighting the critical role of an often-omitted prior masking phase. To scale this process without original pre-training data, we introduce a dual strategy combining linear weight merging with a lightweight multi-domain data mixture that mitigates catastrophic forgetting. Finally, we augment our encoders by merging them with specialized causal models, seamlessly transferring modality- and domain-specific capabilities. This open-source recipe, designed for any causal decoder LLM, yields BidirLM, a family of five encoders that outperform alternatives on text, vision, and audio representation benchmarks. |
| title | BidirLM: From Text to Omnimodal Bidirectional Encoders by Adapting and Composing Causal LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.02045 |