Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916035666378752 |
|---|---|
| author | Myntti, Amanda Kanerva, Jenna Laippala, Veronika Ginter, Filip |
| author_facet | Myntti, Amanda Kanerva, Jenna Laippala, Veronika Ginter, Filip |
| contents | In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks spanning four diverse task categories (retrieval, bitext mining, pair classification, and summarization) in both English and multilingual settings, and reveal that nearest-neighbor overlap and magnitude differences in independent component analysis (ICA) between paired text instances strongly correlate (even up to 0.97) with performance on the given task. Ultimately, we show that embedding tasks display varying degrees of linearity and reliance on retention of local information. Our results further the understanding of embeddings, their relation to model performance, and shed light on possible future training objectives and optimizing conditional embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_22202 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance Myntti, Amanda Kanerva, Jenna Laippala, Veronika Ginter, Filip Computation and Language In this paper, we show that high-performing embedding models organize their embedding spaces in a consistent way. We evaluate 25 contemporary embedding models on five MTEB tasks spanning four diverse task categories (retrieval, bitext mining, pair classification, and summarization) in both English and multilingual settings, and reveal that nearest-neighbor overlap and magnitude differences in independent component analysis (ICA) between paired text instances strongly correlate (even up to 0.97) with performance on the given task. Ultimately, we show that embedding tasks display varying degrees of linearity and reliance on retention of local information. Our results further the understanding of embeddings, their relation to model performance, and shed light on possible future training objectives and optimizing conditional embeddings. |
| title | Structure Retention in Embedding Spaces as a Predictor of Benchmark Performance |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.22202 |