Imagen de Portada

Efficient Continual Pre-training of LLMs for Low-resource Languages

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Nag, Arijit, Chakrabarti, Soumen, Mukherjee, Animesh, Ganguly, Niloy
Formato:	Preprint
Publicado:	2024
Materias:	Computation and Language Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs
por: Nag, Arijit, et al.
Publicado: (2024)

Adapting Multilingual LLMs to Low-Resource Languages using Continued Pre-training and Synthetic Corpus
por: Joshi, Raviraj, et al.
Publicado: (2024)

$FastDoc$: Domain-Specific Fast Continual Pre-training Technique using Document-Level Metadata and Taxonomy
por: Nandy, Abhilash, et al.
Publicado: (2023)

Parameter-Efficient Instruction Tuning of Large Language Models For Extreme Financial Numeral Labelling
por: Khatuya, Subhendu, et al.
Publicado: (2024)

How to think step-by-step: A mechanistic understanding of chain-of-thought reasoning
por: Dutta, Subhabrata, et al.
Publicado: (2024)

MediSwift: Efficient Sparse Pre-trained Biomedical Language Models
por: Thangarasa, Vithursan, et al.
Publicado: (2024)

Efficient Knowledge Probing of Large Language Models by Adapting Pre-trained Embeddings
por: Sharma, Kartik, et al.
Publicado: (2025)

MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization
por: Balde, Gunjan, et al.
Publicado: (2024)

Using Pre-trained LLMs for Multivariate Time Series Forecasting
por: Wolff, Malcolm L., et al.
Publicado: (2025)

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models
por: Gu, Jiawei, et al.
Publicado: (2024)

AlpaPICO: Extraction of PICO Frames from Clinical Trial Documents Using LLMs
por: Ghosh, Madhusudan, et al.
Publicado: (2024)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
por: Ibrahim, Adam, et al.
Publicado: (2024)

Order-Based Pre-training Strategies for Procedural Text Understanding
por: Nandy, Abhilash, et al.
Publicado: (2024)

Hadamard Adapter: An Extreme Parameter-Efficient Adapter Tuning Method for Pre-trained Language Models
por: Chen, Yuyan, et al.
Publicado: (2024)

SpacTor-T5: Pre-training T5 Models with Span Corruption and Replaced Token Detection
por: Ye, Ke, et al.
Publicado: (2024)

Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
por: Nakamura, Taishi, et al.
Publicado: (2024)

DEPT: Decoupled Embeddings for Pre-training Language Models
por: Iacob, Alex, et al.
Publicado: (2024)

Model Merging in Pre-training of Large Language Models
por: Li, Yunshui, et al.
Publicado: (2025)

Fine Tuning Methods for Low-resource Languages
por: Bakkenes, Tim, et al.
Publicado: (2025)

Label-semantics Aware Generative Approach for Domain-Agnostic Multilabel Classification
por: Khatuya, Subhendu, et al.
Publicado: (2025)

Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali
por: Duwal, Sharad, et al.
Publicado: (2024)

Making Pre-trained Language Models Great on Tabular Prediction
por: Yan, Jiahuan, et al.
Publicado: (2024)

Unleashing the Reasoning Potential of Pre-trained LLMs by Critique Fine-Tuning on One Problem
por: Wang, Yubo, et al.
Publicado: (2025)

The Dark Side of the Language: Pre-trained Transformers in the DarkNet
por: Ranaldi, Leonardo, et al.
Publicado: (2022)

Exploiting Vocabulary Frequency Imbalance in Language Model Pre-training
por: Chung, Woojin, et al.
Publicado: (2025)

PaPaformer: Language Model from Pre-trained Parallel Paths
por: Tapaninaho, Joonas, et al.
Publicado: (2025)

Learn or Recall? Revisiting Incremental Learning with Pre-trained Language Models
por: Zheng, Junhao, et al.
Publicado: (2023)

Thinking Augmented Pre-training
por: Wang, Liang, et al.
Publicado: (2025)

Whispering in Amharic: Fine-tuning Whisper for Low-resource Language
por: Gete, Dawit Ketema, et al.
Publicado: (2025)

Perplexity-Aware Data Scaling Law: Perplexity Landscapes Predict Performance for Continual Pre-training
por: Liu, Lei, et al.
Publicado: (2025)

Fine-Tuning Pre-trained Language Models to Detect In-Game Trash Talks
por: Fesalbon, Daniel, et al.
Publicado: (2024)

Pre-trained Large Language Models Use Fourier Features to Compute Addition
por: Zhou, Tianyi, et al.
Publicado: (2024)

HLAT: High-quality Large Language Model Pre-trained on AWS Trainium
por: Fan, Haozheng, et al.
Publicado: (2024)

Structural Pruning of Pre-trained Language Models via Neural Architecture Search
por: Klein, Aaron, et al.
Publicado: (2024)

MLKD-BERT: Multi-level Knowledge Distillation for Pre-trained Language Models
por: Zhang, Ying, et al.
Publicado: (2024)

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
por: Zhong, Zexuan, et al.
Publicado: (2024)

PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training
por: Yi, Rongjie, et al.
Publicado: (2024)

Not How Many, But Which: Parameter Placement in Low-Rank Adaptation
por: Sehanobish, Arijit, et al.
Publicado: (2026)

Domain-Adaptive Continued Pre-Training of Small Language Models
por: Faroz, Salman
Publicado: (2025)

Continual Pre-training of MoEs: How robust is your router?
por: Thérien, Benjamin, et al.
Publicado: (2025)