Copertina

Identifying Semantic Induction Heads to Understand In-Context Learning

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Ren, Jie, Guo, Qipeng, Yan, Hang, Liu, Dongrui, Zhang, Quanshi, Qiu, Xipeng, Lin, Dahua
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Scaling Laws of RoPE-based Extrapolation
di: Liu, Xiaoran, et al.
Pubblicazione: (2023)

ReAttention: Training-Free Infinite Context with Finite Attention Scope
di: Liu, Xiaoran, et al.
Pubblicazione: (2024)

Balanced Data Sampling for Language Model Training with Clustering
di: Shao, Yunfan, et al.
Pubblicazione: (2024)

On the Emergence of Induction Heads for In-Context Learning
di: Musat, Tiberiu, et al.
Pubblicazione: (2025)

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE
di: Zeng, Zhiyuan, et al.
Pubblicazione: (2024)

Towards Economical Inference: Enabling DeepSeek's Multi-Head Latent Attention in Any Transformer-based LLMs
di: Ji, Tao, et al.
Pubblicazione: (2025)

Temporal Dependencies in In-Context Learning: The Role of Induction Heads
di: Bajaj, Anooshka, et al.
Pubblicazione: (2026)

What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practices
di: Chen, Zhi, et al.
Pubblicazione: (2024)

Towards the Dynamics of a DNN Learning Symbolic Interactions
di: Ren, Qihan, et al.
Pubblicazione: (2024)

Entropy-Gradient Inversion: Moving Toward Internal Mechanism of Large Reasoning Models
di: Yang, Junyao, et al.
Pubblicazione: (2026)

Beyond Induction Heads: In-Context Meta Learning Induces Multi-Phase Circuit Emergence
di: Minegishi, Gouki, et al.
Pubblicazione: (2025)

Context-level Language Modeling by Learning Predictive Context Embeddings
di: Dai, Beiya, et al.
Pubblicazione: (2025)

Emergent Structured Representations Support Flexible In-Context Inference in Large Language Models
di: Xu, Ningyu, et al.
Pubblicazione: (2026)

LongWanjuan: Towards Systematic Measurement for Long Text Quality
di: Lv, Kai, et al.
Pubblicazione: (2024)

In-Context Learning in Speech Language Models: Analyzing the Role of Acoustic Features, Linguistic Structure, and Induction Heads
di: Pouw, Charlotte, et al.
Pubblicazione: (2026)

SIM-CoT: Supervised Implicit Chain-of-Thought
di: Wei, Xilin, et al.
Pubblicazione: (2025)

Data-free Weight Compress and Denoise for Large Language Models
di: Peng, Runyu, et al.
Pubblicazione: (2024)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
di: Peng, Runyu, et al.
Pubblicazione: (2026)

Unveiling the Misuse Potential of Base Large Language Models via In-Context Learning
di: Wang, Xiao, et al.
Pubblicazione: (2024)

LongSafety: Enhance Safety for Long-Context LLMs
di: Huang, Mianqiu, et al.
Pubblicazione: (2024)

Flames: Benchmarking Value Alignment of LLMs in Chinese
di: Huang, Kexin, et al.
Pubblicazione: (2023)

Leveraging Grammar Induction for Language Understanding and Generation
di: Kai, Jushi, et al.
Pubblicazione: (2024)

Timely Machine: Awareness of Time Makes Test-Time Scaling Agentic
di: Ma, Yichuan, et al.
Pubblicazione: (2026)

Mousse: Rectifying the Geometry of Muon with Curvature-Aware Preconditioning
di: Zhang, Yechen, et al.
Pubblicazione: (2026)

Quantifying In-Context Reasoning Effects and Memorization Effects in LLMs
di: Lou, Siyu, et al.
Pubblicazione: (2024)

Synergetic Event Understanding: A Collaborative Approach to Cross-Document Event Coreference Resolution with Large Language Models
di: Min, Qingkai, et al.
Pubblicazione: (2024)

Understanding the Role of LLMs in Multimodal Evaluation Benchmarks
di: Jiang, Botian, et al.
Pubblicazione: (2024)

Induction Head Toxicity Mechanistically Explains Repetition Curse in Large Language Models
di: Wang, Shuxun, et al.
Pubblicazione: (2025)

AdaLomo: Low-memory Optimization with Adaptive Learning Rate
di: Lv, Kai, et al.
Pubblicazione: (2023)

Evaluating the Correctness of Inference Patterns Used by LLMs for Judgment
di: Chen, Lu, et al.
Pubblicazione: (2024)

Revisiting Generalization Power of a DNN in Terms of Symbolic Interactions
di: Cheng, Lei, et al.
Pubblicazione: (2025)

FoundaBench: Evaluating Chinese Fundamental Knowledge Capabilities of Large Language Models
di: Li, Wei, et al.
Pubblicazione: (2024)

HERMES: KV Cache as Hierarchical Memory for Efficient Streaming Video Understanding
di: Zhang, Haowei, et al.
Pubblicazione: (2026)

GAOKAO-MM: A Chinese Human-Level Benchmark for Multimodal Models Evaluation
di: Zong, Yi, et al.
Pubblicazione: (2024)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
di: Lu, Yi, et al.
Pubblicazione: (2024)

Enhancing RAG Efficiency with Adaptive Context Compression
di: Guo, Shuyu, et al.
Pubblicazione: (2025)

RvB: Automating AI System Hardening via Iterative Red-Blue Games
di: Huang, Lige, et al.
Pubblicazione: (2026)

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections
di: Wang, Bo, et al.
Pubblicazione: (2025)

Alignment for Honesty
di: Yang, Yuqing, et al.
Pubblicazione: (2023)

Agent Alignment in Evolving Social Norms
di: Li, Shimin, et al.
Pubblicazione: (2024)