Image de couverture de livre

Learning Dynamics of Meta-Learning in Small Model Pretraining

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Africa, David Demitri, Weiss, Yuval, Buttery, Paula, Martinez, Richard Diehl
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Investigating ReLoRA: Effects on the Learning Dynamics of Small Language Models
par: Weiss, Yuval, et autres
Publié: (2025)

Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages
par: Africa, David Demitri, et autres
Publié: (2025)

Pico: A Modular Framework for Hypothesis-Driven Small Language Model Research
par: Martinez, Richard Diehl, et autres
Publié: (2025)

Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies
par: Salhan, Suchir, et autres
Publié: (2024)

LURE: Live-Usage Replay Evaluations for Reducing Evaluation Awareness
par: Ivanov, Igor, et autres
Publié: (2026)

Steering Awareness: Detecting Activation Steering from Within
par: Rivera, Joshua Fonseca, et autres
Publié: (2025)

Consistency Training while Mitigating Obfuscation via Rate Matching
par: Imran, Sohaib, et autres
Publié: (2026)

Tending Towards Stability: Convergence Challenges in Small Language Models
par: Martinez, Richard Diehl, et autres
Publié: (2024)

Looking to Learn: Token-wise Dynamic Gating for Low-Resource Vision-Language Modelling
par: Ganescu, Bianca-Mihaela, et autres
Publié: (2025)

Bias Dynamics in BabyLMs: Towards a Compute-Efficient Sandbox for Democratising Pre-Training Debiasing
par: Trhlik, Filip, et autres
Publié: (2026)

No Answer Needed: Predicting LLM Answer Accuracy from Question-Only Linear Probes
par: Cencerrado, Iván Vicente Moreno, et autres
Publié: (2025)

Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
par: Tan, Daniel, et autres
Publié: (2025)

Identifying a Circuit for Verb Conjugation in GPT-2
par: Africa, David Demitri
Publié: (2025)

Batayan: A Filipino NLP benchmark for evaluating Large Language Models
par: Montalan, Jann Railey, et autres
Publié: (2025)

Does Self-Evaluation Enable Wireheading in Language Models?
par: Africa, David Demitri, et autres
Publié: (2025)

Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
par: Tice, Cameron, et autres
Publié: (2026)

What is the Best Sequence Length for BABYLM?
par: Salhan, Suchir, et autres
Publié: (2025)

Discovering Knowledge-Critical Subnetworks in Pretrained Language Models
par: Bayazit, Deniz, et autres
Publié: (2023)

Pretrained LLMs Learn Multiple Types of Uncertainty
par: Cohen, Roi, et autres
Publié: (2025)

Beyond Random Sampling: Efficient Language Model Pretraining via Curriculum Learning
par: Zhang, Yang, et autres
Publié: (2025)

Mitigating Frequency Bias and Anisotropy in Language Model Pre-Training with Syntactic Smoothing
par: Martinez, Richard Diehl, et autres
Publié: (2024)

From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes
par: Goriely, Zébulon, et autres
Publié: (2024)

Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review
par: Prakriya, Neha, et autres
Publié: (2024)

MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining
par: Dong, Haoyu, et autres
Publié: (2025)

AI Managed Emergency Documentation with a Pretrained Model
par: Menzies, David, et autres
Publié: (2024)

The Tokenization Bottleneck: How Vocabulary Extension Improves Chemistry Representation Learning in Pretrained Language Models
par: Kalamkar, Prathamesh, et autres
Publié: (2025)

Reanalyzing L2 Preposition Learning with Bayesian Mixed Effects and a Pretrained Language Model
par: Prange, Jakob, et autres
Publié: (2023)

Small Vocabularies, Big Gains: Pretraining and Tokenization in Time Series Models
par: Roger, Alexis, et autres
Publié: (2025)

Learning to Learn from Language Feedback with Social Meta-Learning
par: Cook, Jonathan, et autres
Publié: (2026)

Dynamic Masking Rate Schedules for MLM Pretraining
par: Ankner, Zachary, et autres
Publié: (2023)

Beyond Repetition: Text Simplification and Curriculum Learning for Data-Constrained Pretraining
par: Roque, Matthew Theodore, et autres
Publié: (2025)

Meta-aware Learning in text-to-SQL Large Language Model
par: Zhang, Wenda
Publié: (2025)

Enhancing Vaccine Safety Surveillance: Extracting Vaccine Mentions from Emergency Department Triage Notes Using Fine-Tuned Large Language Models
par: Khademi, Sedigh, et autres
Publié: (2025)

Knowledge Graphs are Implicit Reward Models: Path-Derived Signals Enable Compositional Reasoning
par: Kansal, Yuval, et autres
Publié: (2026)

Multilingual Pretraining for Pixel Language Models
par: Kesen, Ilker, et autres
Publié: (2025)

Once Upon a Time: Interactive Learning for Storytelling with Small Language Models
par: Martins, Jonas Mayer, et autres
Publié: (2025)

Pretrained Generative Language Models as General Learning Frameworks for Sequence-Based Tasks
par: Fauber, Ben
Publié: (2024)

Meta-RTL: Reinforcement-Based Meta-Transfer Learning for Low-Resource Commonsense Reasoning
par: Fu, Yu, et autres
Publié: (2024)

Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance
par: Patil, Nirvan, et autres
Publié: (2025)

Transfer Learning for Automated Feedback Generation on Small Datasets
par: Morris, Oscar
Publié: (2025)