Cover Image

Context-Parametric Inversion: Why Instruction Finetuning Can Worsen Context Reliance

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Goyal, Sachin, Baek, Christina, Kolter, J. Zico, Raghunathan, Aditi
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Why is SAM Robust to Label Noise?
by: Baek, Christina, et al.
Published: (2024)

T-MARS: Improving Visual Representations by Circumventing Text Feature Learning
by: Maini, Pratyush, et al.
Published: (2023)

Base Models Look Human To AI Detectors
by: Xu, Yixuan Even, et al.
Published: (2026)

Mode-Conditioning Unlocks Superior Test-Time Scaling
by: Wu, Chen Henry, et al.
Published: (2025)

Understanding Finetuning for Factual Knowledge Extraction
by: Ghosal, Gaurav, et al.
Published: (2024)

Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnostic
by: Goyal, Sachin, et al.
Published: (2024)

Weight Ensembling Improves Reasoning in Language Models
by: Dang, Xingyu, et al.
Published: (2025)

Test-Time Adaptation Induces Stronger Accuracy and Agreement-on-the-Line
by: Kim, Eungyeup, et al.
Published: (2023)

Predicting the Performance of Foundation Models via Agreement-on-the-Line
by: Saxena, Rahul, et al.
Published: (2024)

Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
by: Watts, Ishaan, et al.
Published: (2026)

FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers
by: Williams, Joshua Nathaniel, et al.
Published: (2024)

Predicting the Performance of Black-box LLMs through Follow-up Queries
by: Sam, Dylan, et al.
Published: (2025)

Massive Activations in Large Language Models
by: Sun, Mingjie, et al.
Published: (2024)

Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMs
by: Zhong, Ziqian, et al.
Published: (2025)

Mimetic Initialization Helps State Space Models Learn to Recall
by: Trockman, Asher, et al.
Published: (2024)

A Simple and Effective Pruning Approach for Large Language Models
by: Sun, Mingjie, et al.
Published: (2023)

Looking beyond the next token
by: Thankaraj, Abitha, et al.
Published: (2025)

Forcing Diffuse Distributions out of Language Models
by: Zhang, Yiming, et al.
Published: (2024)

Finetuning CLIP to Reason about Pairwise Differences
by: Sam, Dylan, et al.
Published: (2024)

TOFU: A Task of Fictitious Unlearning for LLMs
by: Maini, Pratyush, et al.
Published: (2024)

Rethinking LLM Memorization through the Lens of Adversarial Compression
by: Schwarzschild, Avi, et al.
Published: (2024)

ImpossibleBench: Measuring LLMs' Propensity of Exploiting Test Cases
by: Zhong, Ziqian, et al.
Published: (2025)

Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters
by: Li, Kevin Y., et al.
Published: (2024)

Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learning
by: Xu, Yixuan Even, et al.
Published: (2025)

Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
by: Gupta, Prakhar, et al.
Published: (2026)

In-Context Learning through the Bayesian Prism
by: Panwar, Madhur, et al.
Published: (2023)

Improving Sparse Memory Finetuning
by: Goyal, Satyam, et al.
Published: (2026)

Self-Trained Verification for Training- and Test-Time Self-Improvement
by: Wu, Chen Henry, et al.
Published: (2026)

Understanding Catastrophic Forgetting in Language Models via Implicit Inference
by: Kotha, Suhas, et al.
Published: (2023)

Quantifying the Plausibility of Context Reliance in Neural Machine Translation
by: Sarti, Gabriele, et al.
Published: (2023)

When Should We Introduce Safety Interventions During Pretraining?
by: Sam, Dylan, et al.
Published: (2026)

Mitigating Bias in RAG: Controlling the Embedder
by: Kim, Taeyoun, et al.
Published: (2025)

Testing the Limits of Jailbreaking Defenses with the Purple Problem
by: Kim, Taeyoun, et al.
Published: (2024)

Diffusion Language Models Can Perform Many Tasks with Scaling and Instruction-Finetuning
by: Ye, Jiasheng, et al.
Published: (2023)

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
by: Vasudeva, Bhavya, et al.
Published: (2026)

Revisiting In-Context Learning with Long Context Language Models
by: Baek, Jinheon, et al.
Published: (2024)

LoRA Users Beware: A Few Spurious Tokens Can Manipulate Your Finetuned Model
by: Salles, Marcel Mateos, et al.
Published: (2025)

AcceleratedLiNGAM: Learning Causal DAGs at the speed of GPUs
by: Akinwande, Victor, et al.
Published: (2024)

Can Language Models Compose Skills In-Context?
by: Liu, Zidong, et al.
Published: (2025)

On the Loss of Context-awareness in General Instruction Fine-tuning
by: Wang, Yihan, et al.
Published: (2024)