ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918191325773824 |
|---|---|
| author | Goldman, Omer Shaham, Uri Malkin, Dan Eiger, Sivan Hassidim, Avinatan Matias, Yossi Maynez, Joshua Gilady, Adi Mayrav Riesa, Jason Rijhwani, Shruti Rimell, Laura Szpektor, Idan Tsarfaty, Reut Eyal, Matan |
| author_facet | Goldman, Omer Shaham, Uri Malkin, Dan Eiger, Sivan Hassidim, Avinatan Matias, Yossi Maynez, Joshua Gilady, Adi Mayrav Riesa, Jason Rijhwani, Shruti Rimell, Laura Szpektor, Idan Tsarfaty, Reut Eyal, Matan |
| contents | To achieve equitable performance across languages, large language models (LLMs) must be able to abstract knowledge beyond the language in which it was learnt. However, the current literature lacks reliable ways to measure LLMs' capability of such cross-lingual knowledge transfer. To that end, we present ECLeKTic, a multilingual closed-book QA dataset that Evaluates Cross-Lingual Knowledge Transfer in a simple, black-box manner. Concretely, we used the presence and absence of Wikipedia articles in 12 languages to detect pieces of information that were likely available during pre-training in one of the languages but not in the others. We curate ECLeKTic as a set of fact-seeking questions over this kind of information, in all the different languages. Therefore, in order to solve ECLeKTic the model is required to transfer knowledge between languages. We evaluated 8 LLMs and showed that current SOTA models struggle to effectively share knowledge across languages, even if they can predict the answer for questions in the language in which the knowledge was acquired. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_21228 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer Goldman, Omer Shaham, Uri Malkin, Dan Eiger, Sivan Hassidim, Avinatan Matias, Yossi Maynez, Joshua Gilady, Adi Mayrav Riesa, Jason Rijhwani, Shruti Rimell, Laura Szpektor, Idan Tsarfaty, Reut Eyal, Matan Computation and Language Artificial Intelligence To achieve equitable performance across languages, large language models (LLMs) must be able to abstract knowledge beyond the language in which it was learnt. However, the current literature lacks reliable ways to measure LLMs' capability of such cross-lingual knowledge transfer. To that end, we present ECLeKTic, a multilingual closed-book QA dataset that Evaluates Cross-Lingual Knowledge Transfer in a simple, black-box manner. Concretely, we used the presence and absence of Wikipedia articles in 12 languages to detect pieces of information that were likely available during pre-training in one of the languages but not in the others. We curate ECLeKTic as a set of fact-seeking questions over this kind of information, in all the different languages. Therefore, in order to solve ECLeKTic the model is required to transfer knowledge between languages. We evaluated 8 LLMs and showed that current SOTA models struggle to effectively share knowledge across languages, even if they can predict the answer for questions in the language in which the knowledge was acquired. |
| title | ECLeKTic: a Novel Challenge Set for Evaluation of Cross-Lingual Knowledge Transfer |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.21228 |