Cover Image

Parallel Structures in Pre-training Data Yield In-Context Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Yanda, Zhao, Chen, Yu, Zhou, McKeown, Kathleen, He, He
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

On the Relation between Sensitivity and Accuracy in In-context Learning
by: Chen, Yanda, et al.
Published: (2022)

Getting Serious about Humor: Crafting Humor Datasets with Unfunny Large Language Models
by: Horvitz, Zachary, et al.
Published: (2024)

Counterfactual Simulatability of LLM Explanations for Generation Tasks
by: Limpijankit, Marvin, et al.
Published: (2025)

ModelingAgent: Bridging LLMs and Mathematical Modeling for Real-World Challenges
by: Qian, Cheng, et al.
Published: (2025)

Social Orientation: A New Feature for Dialogue Analysis
by: Morrill, Todd, et al.
Published: (2024)

Enhancing Multimodal Affective Analysis with Learned Live Comment Features
by: Deng, Zhaoyuan, et al.
Published: (2024)

Towards Consistent Natural-Language Explanations via Explanation-Consistency Finetuning
by: Chen, Yanda, et al.
Published: (2024)

Exploring Chain-of-Thought Reasoning for Steerable Pluralistic Alignment
by: Zhang, Yunfan, et al.
Published: (2025)

Estimating Tail Risks in Language Model Output Distributions
by: Angell, Rico, et al.
Published: (2026)

From $P(y|x)$ to $P(y)$: Investigating Reinforcement Learning in Pre-train Space
by: Tan, Yuqiao, et al.
Published: (2026)

MemGuard: Preventing Memory Contamination in Long-Term Memory-Augmented Large Language Models
by: Ha, Hyeonjeong, et al.
Published: (2026)

Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning
by: Xia, Mengzhou, et al.
Published: (2023)

Data Mixing Agent: Learning to Re-weight Domains for Continual Pre-training
by: Yang, Kailai, et al.
Published: (2025)

LiveNewsBench: Evaluating LLM Web Search Capabilities with Freshly Curated News
by: Zhang, Yunfan, et al.
Published: (2026)

ParaGuide: Guided Diffusion Paraphrasers for Plug-and-Play Textual Style Transfer
by: Horvitz, Zachary, et al.
Published: (2023)

Factuality on Demand: Controlling the Factuality-Informativeness Trade-off in Text Generation
by: Gong, Ziwei, et al.
Published: (2026)

Investigating Data Contamination for Pre-training Language Models
by: Jiang, Minhao, et al.
Published: (2024)

ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
by: Chen, Tong, et al.
Published: (2025)

Programming Every Example: Lifting Pre-training Data Quality Like Experts at Scale
by: Zhou, Fan, et al.
Published: (2024)

LLaSA: Large Language and Structured Data Assistant
by: Xu, Yao, et al.
Published: (2024)

MemoryPrompt: A Light Wrapper to Improve Context Tracking in Pre-trained Language Models
by: Rakotonirina, Nathanaël Carraz, et al.
Published: (2024)

Arena Learning: Build Data Flywheel for LLMs Post-training via Simulated Chatbot Arena
by: Luo, Haipeng, et al.
Published: (2024)

Comparing Pre-trained Human Language Models: Is it Better with Human Context as Groups, Individual Traits, or Both?
by: Soni, Nikita, et al.
Published: (2024)

Pre-training Limited Memory Language Models with Internal and External Knowledge
by: Zhao, Linxi, et al.
Published: (2025)

Deconstructing Pre-training: Knowledge Attribution Analysis in MoE and Dense Models
by: Wang, Bo, et al.
Published: (2026)

Reinforcement Learning on Pre-Training Data
by: Li, Siheng, et al.
Published: (2025)

StyleDistance: Stronger Content-Independent Style Embeddings with Synthetic Parallel Examples
by: Patel, Ajay, et al.
Published: (2024)

Sparse is Enough in Fine-tuning Pre-trained Large Language Models
by: Song, Weixi, et al.
Published: (2023)

Fair In-Context Learning via Latent Concept Variables
by: Bhaila, Karuna, et al.
Published: (2024)

CASE: Efficient Curricular Data Pre-training for Building Assistive Psychology Expert Models
by: Harne, Sarthak, et al.
Published: (2024)

NuNER: Entity Recognition Encoder Pre-training via LLM-Annotated Data
by: Bogdanov, Sergei, et al.
Published: (2024)

A Regularization-based Transfer Learning Method for Information Extraction via Instructed Graph Decoder
by: Chen, Kedi, et al.
Published: (2024)

Selective Reflection-Tuning: Student-Selected Data Recycling for LLM Instruction-Tuning
by: Li, Ming, et al.
Published: (2024)

Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
by: Yang, Yu, et al.
Published: (2024)

Is Pre-training Truly Better Than Meta-Learning?
by: Miranda, Brando, et al.
Published: (2023)

Recycling the Web: A Method to Enhance Pre-training Data Quality and Quantity for Language Models
by: Nguyen, Thao, et al.
Published: (2025)

Bridging Generative and Discriminative Learning: Few-Shot Relation Extraction via Two-Stage Knowledge-Guided Pre-training
by: Guo, Quanjiang, et al.
Published: (2025)

Towards Compute-Optimal Many-Shot In-Context Learning
by: Golchin, Shahriar, et al.
Published: (2025)

Demystifying Synthetic Data in LLM Pre-training: A Systematic Study of Scaling Laws, Benefits, and Pitfalls
by: Kang, Feiyang, et al.
Published: (2025)

Learning to Hint for Reinforcement Learning
by: Xia, Yu, et al.
Published: (2026)