Image de couverture de livre

Induction Heads as an Essential Mechanism for Pattern Matching in In-context Learning

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Crosbie, Joy, Shutova, Ekaterina
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Self-Alignment: Improving Alignment of Cultural Values in LLMs via In-Context Learning
par: Choenni, Rochelle, et autres
Publié: (2024)

The Echoes of Multilinguality: Tracing Cultural Value Shifts during LM Fine-tuning
par: Choenni, Rochelle, et autres
Publié: (2024)

A framework for annotating and modelling intentions behind metaphor use
par: Michelli, Gianluca, et autres
Publié: (2024)

Density Matrices for Metaphor Understanding
par: Owers, Jay, et autres
Publié: (2024)

How do languages influence each other? Studying cross-lingual data sharing during LM fine-tuning
par: Choenni, Rochelle, et autres
Publié: (2023)

Learning New Tasks from a Few Examples with Soft-Label Prototypes
par: Singh, Avyav Kumar, et autres
Publié: (2022)

Yesterday's News: Benchmarking Multi-Dimensional Out-of-Distribution Generalization of Misinformation Detection Models
par: Verhoeven, Ivo, et autres
Publié: (2024)

Metaphor Understanding Challenge Dataset for LLMs
par: Tong, Xiaoyu, et autres
Publié: (2024)

Are LLMs classical or nonmonotonic reasoners? Lessons from generics
par: Leidinger, Alina, et autres
Publié: (2024)

Rethinking Associative Memory Mechanism in Induction Head
par: Wang, Shuo, et autres
Publié: (2024)

On the Evaluation Practices in Multilingual NLP: Can Machine Translation Offer an Alternative to Human Translations?
par: Choenni, Rochelle, et autres
Publié: (2024)

Beyond Words: Exploring Cultural Value Sensitivity in Multimodal Models
par: Yadav, Srishti, et autres
Publié: (2025)

Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning
par: Rajaee, Sara, et autres
Publié: (2025)

On the Emergence of Induction Heads for In-Context Learning
par: Musat, Tiberiu, et autres
Publié: (2025)

The Roots of Performance Disparity in Multilingual Language Models: Intrinsic Modeling Difficulty or Design Choices?
par: Shani, Chen, et autres
Publié: (2026)

Cross-modal Information Flow in Multimodal Large Language Models
par: Zhang, Zhi, et autres
Publié: (2024)

NeuroAda: Activating Each Neuron's Potential for Parameter-Efficient Fine-Tuning
par: Zhang, Zhi, et autres
Publié: (2025)

CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension
par: Zhang, Zhi, et autres
Publié: (2023)

Understanding and Controlling Repetition Neurons and Induction Heads in In-Context Learning
par: Doan, Nhi Hoai, et autres
Publié: (2025)

Hummus: A Dataset of Humorous Multimodal Metaphor Use
par: Tong, Xiaoyu, et autres
Publié: (2025)

Identifying Semantic Induction Heads to Understand In-Context Learning
par: Ren, Jie, et autres
Publié: (2024)

Temporal Dependencies in In-Context Learning: The Role of Induction Heads
par: Bajaj, Anooshka, et autres
Publié: (2026)

Predicting the Emergence of Induction Heads in Language Model Pretraining
par: Aoyama, Tatsuya, et autres
Publié: (2025)

A (More) Realistic Evaluation Setup for Generalisation of Community Models on Malicious Content Detection
par: Verhoeven, Ivo, et autres
Publié: (2024)

Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence
par: Minegishi, Gouki, et autres
Publié: (2025)

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
par: Chen, Siyu, et autres
Publié: (2024)

In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads
par: Pouw, Charlotte, et autres
Publié: (2026)

Interpretable Next-token Prediction via the Generalized Induction Head
par: Kim, Eunji, et autres
Publié: (2024)

Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
par: Wang, Shuxun, et autres
Publié: (2025)

Evaluating Memory Structure in LLM Agents
par: Shutova, Alina, et autres
Publié: (2026)

Induction Signatures Are Not Enough: A Matched-Compute Study of Load-Bearing Structure in In-Context Learning
par: Sabry, Mohammed, et autres
Publié: (2025)

Thinking with Patterns: Breaking the Perceptual Bottleneck in Visual Planning via Pattern Induction
par: Jian, Yichang, et autres
Publié: (2026)

Prompt-MII: Meta-Learning Instruction Induction for LLMs
par: Xiao, Emily, et autres
Publié: (2025)

Learning Evidence of Depression Symptoms via Prompt Induction
par: Bao, Eliseo, et autres
Publié: (2026)

Mechanism of Task-oriented Information Removal in In-context Learning
par: Cho, Hakaze, et autres
Publié: (2025)

Going Beyond Word Matching: Syntax Improves In-context Example Selection for Machine Translation
par: Tang, Chenming, et autres
Publié: (2024)

A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs
par: Shelmanov, Artem, et autres
Publié: (2025)

Learning Beyond Pattern Matching? Assaying Mathematical Understanding in LLMs
par: Guo, Siyuan, et autres
Publié: (2024)

Semi-Supervised Learning for Bilingual Lexicon Induction
par: Garnier, Paul, et autres
Publié: (2024)

Uncertainty-Aware Attention Heads: Efficient Unsupervised Uncertainty Quantification for LLMs
par: Vazhentsev, Artem, et autres
Publié: (2025)