Cover Image

Localizing Task Recognition and Task Learning in In-Context Learning via Attention Head Analysis

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Haolin, Cho, Hakaze, Inoue, Naoya
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learning
by: Yang, Haolin, et al.
Published: (2025)

Task Vectors, Learned Not Extracted: Performance Gains and Mechanistic Insight
by: Yang, Haolin, et al.
Published: (2025)

Mechanism of Task-oriented Information Removal in In-context Learning
by: Cho, Hakaze, et al.
Published: (2025)

StaICC: Standardized Evaluation for Classification Task in In-context Learning
by: Cho, Hakaze, et al.
Published: (2025)

Revisiting In-context Learning Inference Circuit in Large Language Models
by: Cho, Hakaze, et al.
Published: (2024)

Binary Autoencoder for Mechanistic Interpretability of Large Language Models
by: Cho, Hakaze, et al.
Published: (2025)

Affinity and Diversity: A Unified Metric for Demonstration Selection via Internal Representations
by: Kato, Mariko, et al.
Published: (2025)

Mechanistic Fine-tuning for In-context Learning
by: Cho, Hakaze, et al.
Published: (2025)

Token-based Decision Criteria Are Suboptimal in In-context Learning
by: Cho, Hakaze, et al.
Published: (2024)

Understanding Token Probability Encoding in Output Embeddings
by: Cho, Hakaze, et al.
Published: (2024)

Label Words as Local Task Vectors in In-Context Learning
by: Zheng, Bowen, et al.
Published: (2024)

Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
by: Wang, Xiaolei, et al.
Published: (2024)

Task-KV: Task-aware KV Cache Optimization via Semantic Differentiation of Attention Heads
by: He, Xingyang, et al.
Published: (2025)

Task Vector Geometry Underlies Dual Modes of Task Inference in Transformers
by: Yan, Hao, et al.
Published: (2026)

Learning Task Representations from In-Context Learning
by: Saglam, Baturay, et al.
Published: (2025)

Measuring Intrinsic Dimension of Token Embeddings
by: Kataiwa, Takuya, et al.
Published: (2025)

Task-Oriented Dialogue with In-Context Learning
by: Bocklisch, Tom, et al.
Published: (2024)

Which Attention Heads Matter for In-Context Learning?
by: Yin, Kayo, et al.
Published: (2025)

MuDAF: Long-Context Multi-Document Attention Focusing through Contrastive Learning on Attention Heads
by: Liu, Weihao, et al.
Published: (2025)

Does In-Context Learning Really Learn? Rethinking How Large Language Models Respond and Solve Tasks via In-Context Learning
by: Long, Quanyu, et al.
Published: (2024)

CrossICL: Cross-Task In-Context Learning via Unsupervised Demonstration Transfer
by: Gao, Jinglong, et al.
Published: (2025)

Effective Multi-Task Learning for Biomedical Named Entity Recognition
by: Ruano, João, et al.
Published: (2025)

The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentation
by: Kahardipraja, Patrick, et al.
Published: (2025)

NoisyICL: A Little Noise in Model Parameters Calibrates In-context Learning
by: Zhao, Yufeng, et al.
Published: (2024)

TaskComplexity: A Dataset for Task Complexity Classification with In-Context Learning, FLAN-T5 and GPT-4o Benchmarks
by: Rasheed, Areeg Fahad, et al.
Published: (2024)

Supervised Fine-Tuning Achieve Rapid Task Adaption Via Alternating Attention Head Activation Patterns
by: Zhao, Yang, et al.
Published: (2024)

Scoping Review of Active Learning Strategies and their Evaluation Environments for Entity Recognition Tasks
by: Kohl, Philipp, et al.
Published: (2024)

In-Context Transfer Learning: Demonstration Synthesis by Transferring Similar Tasks
by: Wang, Dingzirui, et al.
Published: (2024)

TDR: Task-Decoupled Retrieval with Fine-Grained LLM Feedback for In-Context Learning
by: Chen, Yifu, et al.
Published: (2025)

One Task Vector is not Enough: A Large-Scale Study for In-Context Learning
by: Tikhonov, Pavel, et al.
Published: (2025)

GRaSp: Automatic Example Optimization for In-Context Learning in Low-Data Tasks
by: Bihaug-Frøyland, Simen, et al.
Published: (2026)

S2-Attention: Hardware-Aware Context Sharding Among Attention Heads
by: Lin, Xihui, et al.
Published: (2024)

RRAttention: Dynamic Block Sparse Attention via Per-Head Round-Robin Shifts for Long-Context Inference
by: Liu, Siran, et al.
Published: (2026)

From Unstructured Data to In-Context Learning: Exploring What Tasks Can Be Learned and When
by: Wibisono, Kevin Christian, et al.
Published: (2024)

Task Schema and Binding: A Double Dissociation Study of In-Context Learning
by: Kim, Chaeha
Published: (2025)

Distributional Alignment as a Criterion for Designing Task Vectors in In-Context Learning
by: Kwon, Jihoon, et al.
Published: (2026)

What Do Language Models Learn in Context? The Structured Task Hypothesis
by: Li, Jiaoda, et al.
Published: (2024)

On the Emergence of Induction Heads for In-Context Learning
by: Musat, Tiberiu, et al.
Published: (2025)

Multi-Task Learning with LLMs for Implicit Sentiment Analysis: Data-level and Task-level Automatic Weight Learning
by: Lai, Wenna, et al.
Published: (2024)

A Case Study on Context-Aware Neural Machine Translation with Multi-Task Learning
by: Appicharla, Ramakrishna, et al.
Published: (2024)