Cover Image

A Framework for Quantifying How Pre-Training and Context Benefit In-Context Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Song, Bingqing, Li, Jiaxiang, Wang, Rong, Lu, Songtao, Hong, Mingyi
Format:	Preprint
Published:	2025
Subjects:	Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approach
by: Zhang, Xinnan, et al.
Published: (2025)

Generative Pre-Trained Transformer for Symbolic Regression Base In-Context Reinforcement Learning
by: Li, Yanjie, et al.
Published: (2024)

Memory-Efficient LLM Pretraining via Minimalist Optimizer Design
by: Glentis, Athanasios, et al.
Published: (2025)

Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs
by: Song, Woomin, et al.
Published: (2024)

How Transformers Learn In-Context Recall Tasks? Optimality, Training Dynamics and Generalization
by: Nguyen, Quan, et al.
Published: (2025)

When Demonstrations Meet Generative World Models: A Maximum Likelihood Framework for Offline Inverse Reinforcement Learning
by: Zeng, Siliang, et al.
Published: (2023)

Towards Context-Aware Domain Generalization: Understanding the Benefits and Limits of Marginal Transfer Learning
by: Müller, Jens, et al.
Published: (2023)

Relational In-Context Learning via Synthetic Pre-training with Structural Prior
by: Wang, Yanbo, et al.
Published: (2026)

Scalable Parameter and Memory Efficient Pretraining for LLM: Recent Algorithmic Advances and Benchmarking
by: Glentis, Athanasios, et al.
Published: (2025)

FastGAS: Fast Graph-based Annotation Selection for In-Context Learning
by: Chen, Zihan, et al.
Published: (2024)

Scalable In-Context Q-Learning
by: Liu, Jinmei, et al.
Published: (2025)

Cumulative Hazard Function Based Efficient Multivariate Temporal Point Process Learning
by: Liu, Bingqing
Published: (2024)

Vintix II: Decision Pre-Trained Transformer is a Scalable In-Context Reinforcement Learner
by: Polubarov, Andrei, et al.
Published: (2026)

Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds
by: Wang, Fan, et al.
Published: (2025)

In-Context Compositional Q-Learning for Offline Reinforcement Learning
by: Xu, Qiushui, et al.
Published: (2025)

Context and Diversity Matter: The Emergence of In-Context Learning in World Models
by: Wang, Fan, et al.
Published: (2025)

In-Context Reinforcement Learning via Communicative World Models
by: Martinez-Lopez, Fernando, et al.
Published: (2025)

Scaling Laws and In-Context Learning: A Unified Theoretical Framework
by: Mehta, Sushant, et al.
Published: (2025)

How Few-Shot Examples Add Up: A Causal Decomposition of Function Vectors in In-Context Learning
by: Wang, Entang, et al.
Published: (2026)

SparseBalance: Load-Balanced Long Context Training with Dynamic Sparse Attention
by: Xu, Hongtao, et al.
Published: (2026)

Do LLMs Recognize Your Latent Preferences? A Benchmark for Latent Information Discovery in Personalized Interaction
by: Tsaknakis, Ioannis, et al.
Published: (2025)

SlimPipe: Memory-Thrifty and Efficient Pipeline Parallelism for Long-Context LLM Training
by: Li, Zhouyang, et al.
Published: (2025)

Learning to Select In-Context Demonstration Preferred by Large Language Model
by: Zhang, Zheng, et al.
Published: (2025)

Parallel Structures in Pre-training Data Yield In-Context Learning
by: Chen, Yanda, et al.
Published: (2024)

Towards Monotonic Improvement in In-Context Reinforcement Learning
by: Zhang, Wenhao, et al.
Published: (2025)

Technical Debt in In-Context Learning: Diminishing Efficiency in Long Context
by: Joo, Taejong, et al.
Published: (2025)

Reflective Context Learning: Studying the Optimization Primitives of Context Space
by: Vassilyev, Nikita, et al.
Published: (2026)

Feature Alignment: Rethinking Efficient Active Learning via Proxy in the Context of Pre-trained Models
by: Wen, Ziting, et al.
Published: (2024)

Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning
by: Zhang, Kehao, et al.
Published: (2026)

Mixture-of-Experts Meets In-Context Reinforcement Learning
by: Wu, Wenhao, et al.
Published: (2025)

Benchmarking General-Purpose In-Context Learning
by: Wang, Fan, et al.
Published: (2024)

Context Tuning for In-Context Optimization
by: Lu, Jack, et al.
Published: (2025)

Fast Training Dataset Attribution via In-Context Learning
by: Fotouhi, Milad, et al.
Published: (2024)

In-Context Reinforcement Learning through Bayesian Fusion of Context and Value Prior
by: Berkes, Anaïs, et al.
Published: (2026)

Learning Dynamics in Continual Pre-Training for Large Language Models
by: Wang, Xingjin, et al.
Published: (2025)

Training Long-Context LLMs Efficiently via Chunk-wise Optimization
by: Li, Wenhao, et al.
Published: (2025)

Demystifying Manifold Constraints in LLM Pre-training
by: An, Kang, et al.
Published: (2026)

Document-Level In-Context Few-Shot Relation Extraction via Pre-Trained Language Models
by: Ozyurt, Yilmazcan, et al.
Published: (2023)

Understanding the Dynamics of Demonstration Conflict in In-Context Learning
by: Jiao, Difan, et al.
Published: (2026)

Muon Outperforms Adam in Tail-End Associative Memory Learning
by: Wang, Shuche, et al.
Published: (2025)