Continual-learning for Modelling Low-Resource Languages from Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917192393555968 |
|---|---|
| author | K, Santosh Srinath Somani, Mudit Padala, Varun Reddy Upadhyay, Prajna Devi Das, Abhijit |
| author_facet | K, Santosh Srinath Somani, Mudit Padala, Varun Reddy Upadhyay, Prajna Devi Das, Abhijit |
| contents | Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_05874 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Continual-learning for Modelling Low-Resource Languages from Large Language Models K, Santosh Srinath Somani, Mudit Padala, Varun Reddy Upadhyay, Prajna Devi Das, Abhijit Computation and Language Artificial Intelligence Modelling a language model for a multi-lingual scenario includes several potential challenges, among which catastrophic forgetting is the major challenge. For example, small language models (SLM) built for low-resource languages by adapting large language models (LLMs) pose the challenge of catastrophic forgetting. This work proposes to employ a continual learning strategy using parts-of-speech (POS)-based code-switching along with a replay adapter strategy to mitigate the identified gap of catastrophic forgetting while training SLM from LLM. Experiments conducted on vision language tasks such as visual question answering and language modelling task exhibits the success of the proposed architecture. |
| title | Continual-learning for Modelling Low-Resource Languages from Large Language Models |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.05874 |