Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.01592 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915318258991104 |
|---|---|
| author | Elshabrawy, Ahmed Nguyen, Thanh-Nhi Kang, Yeeun Feng, Lihan Jain, Annant Shaikh, Faadil Abdullah Mansurov, Jonibek Imam, Mohamed Fazli Mohamed Ortiz-Barajas, Jesus-German Chevi, Rendi Aji, Alham Fikri |
| author_facet | Elshabrawy, Ahmed Nguyen, Thanh-Nhi Kang, Yeeun Feng, Lihan Jain, Annant Shaikh, Faadil Abdullah Mansurov, Jonibek Imam, Mohamed Fazli Mohamed Ortiz-Barajas, Jesus-German Chevi, Rendi Aji, Alham Fikri |
| contents | Large Language Models (LLMs) excel in zero-shot and few-shot tasks, but achieving similar performance with encoder-only models like BERT and RoBERTa has been challenging due to their architecture. However, encoders offer advantages such as lower computational and memory costs. Recent work adapts them for zero-shot generalization using Statement Tuning, which reformulates tasks into finite templates. We extend this approach to multilingual NLP, exploring whether encoders can achieve zero-shot cross-lingual generalization and serve as efficient alternatives to memory-intensive LLMs for low-resource languages. Our results show that state-of-the-art encoder models generalize well across languages, rivaling multilingual LLMs while being more efficient. We also analyze multilingual Statement Tuning dataset design, efficiency gains, and language-specific generalization, contributing to more inclusive and resource-efficient NLP models. We release our code and models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_01592 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models Elshabrawy, Ahmed Nguyen, Thanh-Nhi Kang, Yeeun Feng, Lihan Jain, Annant Shaikh, Faadil Abdullah Mansurov, Jonibek Imam, Mohamed Fazli Mohamed Ortiz-Barajas, Jesus-German Chevi, Rendi Aji, Alham Fikri Computation and Language Large Language Models (LLMs) excel in zero-shot and few-shot tasks, but achieving similar performance with encoder-only models like BERT and RoBERTa has been challenging due to their architecture. However, encoders offer advantages such as lower computational and memory costs. Recent work adapts them for zero-shot generalization using Statement Tuning, which reformulates tasks into finite templates. We extend this approach to multilingual NLP, exploring whether encoders can achieve zero-shot cross-lingual generalization and serve as efficient alternatives to memory-intensive LLMs for low-resource languages. Our results show that state-of-the-art encoder models generalize well across languages, rivaling multilingual LLMs while being more efficient. We also analyze multilingual Statement Tuning dataset design, efficiency gains, and language-specific generalization, contributing to more inclusive and resource-efficient NLP models. We release our code and models. |
| title | Statement-Tuning Enables Efficient Cross-lingual Generalization in Encoder-only Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.01592 |