Cover Image

Brewing Knowledge in Context: Distillation Perspectives on In-Context Learning

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Chengye, Liu, Haiyun, Li, Yuanxi
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Short Data, Long Context: Distilling Positional Knowledge in Transformers
by: Huber, Patrick, et al.
Published: (2026)

Latent Context Compilation: Distilling Long Context into Compact Portable Memory
by: Li, Zeju, et al.
Published: (2026)

Towards Better Understanding of In-Context Learning Ability from In-Context Uncertainty Quantification
by: Liu, Shang, et al.
Published: (2024)

UT-ACA: Uncertainty-Triggered Adaptive Context Allocation for Long-Context Inference
by: Zhou, Lang, et al.
Published: (2026)

DeMeVa at LeWiDi-2025: Modeling Perspectives with In-Context Learning and Label Distribution Learning
by: Ignatev, Daniil, et al.
Published: (2025)

ContextRL: Enhancing MLLM's Knowledge Discovery Efficiency with Context-Augmented RL
by: Lu, Xingyu, et al.
Published: (2026)

Core Context Aware Transformers for Long Context Language Modeling
by: Chen, Yaofo, et al.
Published: (2024)

GATES: Self-Distillation under Privileged Context with Consensus Gating
by: Stein, Alex, et al.
Published: (2026)

Fine-Grained Modeling of Narrative Context: A Coherence Perspective via Retrospective Questions
by: Xu, Liyan, et al.
Published: (2024)

Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation
by: Ren, Yuxin, et al.
Published: (2023)

UCS: Estimating Unseen Coverage for Improved In-Context Learning
by: Xin, Jiayi, et al.
Published: (2026)

Sinkhorn Distance Minimization for Knowledge Distillation
by: Cui, Xiao, et al.
Published: (2024)

In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks
by: Wang, Dingzirui, et al.
Published: (2024)

LongReD: Mitigating Short-Text Degradation of Long-Context Large Language Models via Restoration Distillation
by: Dong, Zican, et al.
Published: (2025)

APB: Accelerating Distributed Long-Context Inference by Passing Compressed Context Blocks across GPUs
by: Huang, Yuxiang, et al.
Published: (2025)

What Do Language Models Learn in Context? The Structured Task Hypothesis
by: Li, Jiaoda, et al.
Published: (2024)

SEEKR: Selective Attention-Guided Knowledge Retention for Continual Learning of Large Language Models
by: He, Jinghan, et al.
Published: (2024)

RIPPLECOT: Amplifying Ripple Effect of Knowledge Editing in Language Models via Chain-of-Thought In-Context Learning
by: Zhao, Zihao, et al.
Published: (2024)

Intrinsic Entropy of Context Length Scaling in LLMs
by: Shi, Jingzhe, et al.
Published: (2025)

Learning Task Representations from In-Context Learning
by: Saglam, Baturay, et al.
Published: (2025)

Dynamic Attention-Guided Context Decoding for Mitigating Context Faithfulness Hallucinations in Large Language Models
by: Huang, Yanwen, et al.
Published: (2025)

Knowledge Distillation with Training Wheels
by: Liu, Guanlin, et al.
Published: (2025)

Understanding Emergent In-Context Learning from a Kernel Regression Perspective
by: Han, Chi, et al.
Published: (2023)

Emergence and Effectiveness of Task Vectors in In-Context Learning: An Encoder Decoder Perspective
by: Han, Seungwook, et al.
Published: (2024)

Stronger Random Baselines for In-Context Learning
by: Yauney, Gregory, et al.
Published: (2024)

In-Context Language Learning: Architectures and Algorithms
by: Akyürek, Ekin, et al.
Published: (2024)

In-Context Learning through the Bayesian Prism
by: Panwar, Madhur, et al.
Published: (2023)

ContextCite: Attributing Model Generation to Context
by: Cohen-Wang, Benjamin, et al.
Published: (2024)

Self-Consolidating Language Models: Continual Knowledge Incorporation from Context
by: Wang, Zekun, et al.
Published: (2026)

Is In-Context Learning Learning?
by: de Wynter, Adrian
Published: (2025)

Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learning
by: Liu, Hui, et al.
Published: (2024)

ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models
by: Liang, Manlai, et al.
Published: (2025)

Revisiting In-Context Learning with Long Context Language Models
by: Baek, Jinheon, et al.
Published: (2024)

Uncertainty Quantification for In-Context Learning of Large Language Models
by: Ling, Chen, et al.
Published: (2024)

In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation
by: Chen, Shiqi, et al.
Published: (2024)

Multi-modal Anchor Gated Transformer with Knowledge Distillation for Emotion Recognition in Conversation
by: Li, Jie, et al.
Published: (2025)

Exploring Imbalanced Annotations for Effective In-Context Learning
by: Gao, Hongfu, et al.
Published: (2025)

On the Relationship Between the Choice of Representation and In-Context Learning
by: Marinescu, Ioana, et al.
Published: (2025)

On the Noise Robustness of In-Context Learning for Text Generation
by: Gao, Hongfu, et al.
Published: (2024)

Personalized Adaptation via In-Context Preference Learning
by: Lau, Allison, et al.
Published: (2024)