BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908744104804352 |
|---|---|
| author | Gillin, Taj Lalani, Adam Zhang, Kenneth Salles, Marcel Mateos |
| author_facet | Gillin, Taj Lalani, Adam Zhang, Kenneth Salles, Marcel Mateos |
| contents | Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_00366 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics Gillin, Taj Lalani, Adam Zhang, Kenneth Salles, Marcel Mateos Computation and Language Artificial Intelligence Machine Learning Joint Embedding Predictive Architectures (JEPA) are a novel self supervised training technique that have shown recent promise across domains. We introduce BERT-JEPA (BEPA), a training paradigm that adds a JEPA training objective to BERT-style models, working to combat a collapsed [CLS] embedding space and turning it into a language-agnostic space. This new structure leads to increased performance across multilingual benchmarks. |
| title | BERT-JEPA: Reorganizing CLS Embeddings for Language-Invariant Semantics |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.00366 |