Cover Image

Pre-training a Transformer-Based Generative Model Using a Small Sepedi Dataset

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Ramalepe, Simon P., Modipa, Thipe I., Davel, Marelie H.
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

The evaluation of a code-switched Sepedi-English automatic speech recognition system
by: Phaladi, Amanda, et al.
Published: (2024)

Neural Network-based Vehicular Channel Estimation Performance: Effect of Noise in the Training Set
by: Ngorima, Simbarashe Aldrin, et al.
Published: (2025)

Aligning Pre-trained Models for Spoken Language Translation
by: Sedláček, Šimon, et al.
Published: (2024)

Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization
by: Samragh, Mohammad, et al.
Published: (2024)

PhoneLM:an Efficient and Capable Small Language Model Family through Principled Pre-training
by: Yi, Rongjie, et al.
Published: (2024)

Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
by: Yang, Yu, et al.
Published: (2024)

CDGP: Automatic Cloze Distractor Generation based on Pre-trained Language Model
by: Chiang, Shang-Hsuan, et al.
Published: (2024)

Investigating Data Contamination for Pre-training Language Models
by: Jiang, Minhao, et al.
Published: (2024)

Sequence-to-Sequence Spanish Pre-trained Language Models
by: Araujo, Vladimir, et al.
Published: (2023)

Automated Multi-Language to English Machine Translation Using Generative Pre-Trained Transformers
by: Pelofske, Elijah, et al.
Published: (2024)

Integrating Pre-trained Language Model into Neural Machine Translation
by: Hwang, Soon-Jae, et al.
Published: (2023)

Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?
by: Soni, Nikita, et al.
Published: (2024)

Simple and Scalable Strategies to Continually Pre-train Large Language Models
by: Ibrahim, Adam, et al.
Published: (2024)

Pre-training Limited Memory Language Models with Internal and External Knowledge
by: Zhao, Linxi, et al.
Published: (2025)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

Graph Generative Pre-trained Transformer
by: Chen, Xiaohui, et al.
Published: (2025)

Automated Software Vulnerability Static Code Analysis Using Generative Pre-Trained Transformer Models
by: Pelofske, Elijah, et al.
Published: (2024)

Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
by: Wang, Bo, et al.
Published: (2026)

HiFloat4 Format for Language Model Pre-training on Ascend NPUs
by: Taghian, Mehran, et al.
Published: (2026)

Learning to Rank Chain-of-Thought: Using a Small Model
by: Jiang, Eric Hanchen, et al.
Published: (2025)

Coarse-Tuning for Ad-hoc Document Retrieval Using Pre-trained Language Models
by: Keyaki, Atsushi, et al.
Published: (2024)

Too Big to Think: Capacity, Memorization, and Generalization in Pre-Trained Transformers
by: Barron, Joshua, et al.
Published: (2025)

CASE: Efficient Curricular Data Pre-training for Building Assistive Psychology Expert Models
by: Harne, Sarthak, et al.
Published: (2024)

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
by: Xia, Mengzhou, et al.
Published: (2023)

ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
by: Chen, Tong, et al.
Published: (2025)

SlimQwen: Exploring the Pruning and Distillation in Large MoE Model Pre-training
by: Tang, Shengkun, et al.
Published: (2026)

Evaluating Gender Bias Transfer between Pre-trained and Prompt-Adapted Language Models
by: Mackraz, Natalie, et al.
Published: (2024)

MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models
by: Rakotonirina, Nathanaël Carraz, et al.
Published: (2024)

Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?
by: Sakai, Yusuke, et al.
Published: (2023)

Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models
by: Nguyen, Thao, et al.
Published: (2025)

Parallel Structures in Pre-training Data Yield In-Context Learning
by: Chen, Yanda, et al.
Published: (2024)

Machine Unlearning of Pre-trained Large Language Models
by: Yao, Jin, et al.
Published: (2024)

CoLoR-Filter: Conditional Loss Reduction Filtering for Targeted Language Model Pre-training
by: Brandfonbrener, David, et al.
Published: (2024)

Continual Pre-training of MoEs: How robust is your router?
by: Thérien, Benjamin, et al.
Published: (2025)

Bridging Generative and Discriminative Learning: Few-Shot Relation Extraction via Two-Stage Knowledge-Guided Pre-training
by: Guo, Quanjiang, et al.
Published: (2025)

CaLM: Contrasting Large and Small Language Models to Verify Grounded Generation
by: Hsu, I-Hung, et al.
Published: (2024)

Zero-shot Medical Event Prediction Using a Generative Pre-trained Transformer on Electronic Health Records
by: Redekop, Ekaterina, et al.
Published: (2025)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
by: Yang, Kailai, et al.
Published: (2025)

Aurora-M: Open Source Continual Pre-training for Multilingual Language and Code
by: Nakamura, Taishi, et al.
Published: (2024)

Does simple trump complex? Comparing strategies for adversarial robustness in DNNs
by: Brooks, William, et al.
Published: (2025)