Image de couverture de livre

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Vasudeva, Bhavya, Deora, Puneesh, Bietti, Alberto, Sharan, Vatsal, Thrampoulidis, Christos
Format:	Preprint
Publié:	2026
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

In-Context Occam's Razor: How Transformers Prefer Simpler Hypotheses on the Fly
par: Deora, Puneesh, et autres
Publié: (2025)

How Muon's Spectral Design Benefits Generalization: A Study on Imbalanced Data
par: Vasudeva, Bhavya, et autres
Publié: (2025)

Implicit Bias and Fast Convergence Rates for Self-attention
par: Vasudeva, Bhavya, et autres
Publié: (2024)

Facts in Stats: Impacts of Pretraining Diversity on Language Model Generalization
par: Behnia, Tina, et autres
Publié: (2025)

On the Optimization and Generalization of Multi-head Attention
par: Deora, Puneesh, et autres
Publié: (2023)

Transformers Learn Low Sensitivity Functions: Investigations and Implications
par: Vasudeva, Bhavya, et autres
Publié: (2024)

Latent Concept Disentanglement in Transformer-based Language Models
par: Hong, Guan Zhe, et autres
Publié: (2025)

Understanding Factual Recall in Transformers via Associative Memories
par: Nichani, Eshaan, et autres
Publié: (2024)

The Rich and the Simple: On the Implicit Bias of Adam and SGD
par: Vasudeva, Bhavya, et autres
Publié: (2025)

Implicit Optimization Bias of Next-Token Prediction in Linear Models
par: Thrampoulidis, Christos
Publié: (2024)

Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
par: Fu, Deqing, et autres
Publié: (2023)

Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers
par: Chen, Lei, et autres
Publié: (2024)

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
par: Feng, Jiahai, et autres
Publié: (2024)

Understanding Finetuning for Factual Knowledge Extraction
par: Ghosal, Gaurav, et autres
Publié: (2024)

Learning to Recall with Transformers Beyond Orthogonal Embeddings
par: Vural, Nuri Mert, et autres
Publié: (2026)

Convergent Evolution: How Different Language Models Learn Similar Number Representations
par: Fu, Deqing, et autres
Publié: (2026)

Transformers as Support Vector Machines
par: Tarzanagh, Davoud Ataee, et autres
Publié: (2023)

Pre-trained Large Language Models Use Fourier Features to Compute Addition
par: Zhou, Tianyi, et autres
Publié: (2024)

Implicit Geometry of Next-token Prediction: From Language Sparsity Patterns to Model Representations
par: Zhao, Yize, et autres
Publié: (2024)

Contextual Drag: How Errors in the Context Affect LLM Reasoning
par: Cheng, Yun, et autres
Publié: (2026)

Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers
par: Huang, Yixiao, et autres
Publié: (2025)

Limitations on Accurate, Trusted, Human-level Reasoning
par: Panigrahy, Rina, et autres
Publié: (2025)

Geometric Factual Recall in Transformers
par: Ravfogel, Shauli, et autres
Publié: (2026)

FoNE: Precise Single-Token Number Embeddings via Fourier Features
par: Zhou, Tianyi, et autres
Publié: (2025)

LLM In-Context Recall is Prompt Dependent
par: Machlab, Daniel, et autres
Publié: (2024)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
par: Bethune, Louis, et autres
Publié: (2025)

Memorization Capacity of Multi-Head Attention in Transformers
par: Mahdavi, Sadegh, et autres
Publié: (2023)

Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Reasoning
par: Štorek, Adam, et autres
Publié: (2025)

Ulterior Motives: Detecting Misaligned Reasoning in Continuous Thought Models
par: Ramjee, Sharan
Publié: (2026)

Hierarchical Retrieval: The Geometry and a Pretrain-Finetune Recipe
par: You, Chong, et autres
Publié: (2025)

Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models
par: Ruis, Laura, et autres
Publié: (2024)

Chameleon: A Flexible Data-mixing Framework for Language Model Pretraining and Finetuning
par: Xie, Wanyun, et autres
Publié: (2025)

Compute Optimal Scaling of Skills: Knowledge vs Reasoning
par: Roberts, Nicholas, et autres
Publié: (2025)

Can GPT Redefine Medical Understanding? Evaluating GPT on Biomedical Machine Reading Comprehension
par: Vatsal, Shubham, et autres
Publié: (2024)

Why Loss Re-weighting Works If You Stop Early: Training Dynamics of Unconstrained Features
par: Zhao, Yize, et autres
Publié: (2026)

Supervised Contrastive Representation Learning: Landscape Analysis with Unconstrained Features
par: Behnia, Tina, et autres
Publié: (2024)

ATLAS: Adaptive Transfer Scaling Laws for Multilingual Pretraining, Finetuning, and Decoding the Curse of Multilinguality
par: Longpre, Shayne, et autres
Publié: (2025)

On the Effect of Negative Gradient in Group Relative Deep Reinforcement Optimization
par: Deng, Wenlong, et autres
Publié: (2025)

Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance
par: Goyal, Sachin, et autres
Publié: (2024)

Generalizable and Stable Finetuning of Pretrained Language Models on Low-Resource Texts
par: Somayajula, Sai Ashish, et autres
Publié: (2024)