Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Chen, Lei, Bruna, Joan, Bietti, Alberto
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Compositional Reasoning with Transformers, RNNs, and Chain of Thought
di: Yehudai, Gilad, et al.
Pubblicazione: (2025)

Every Attention Matters: An Efficient Hybrid Architecture for Long-Context Reasoning
di: Ling Team, et al.
Pubblicazione: (2025)

Scaling Laws for Associative Memories
di: Cabannes, Vivien, et al.
Pubblicazione: (2023)

Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
di: Zhu, Kan, et al.
Pubblicazione: (2025)

FROST: Filtering Reasoning Outliers with Attention for Efficient Reasoning
di: Luo, Haozheng, et al.
Pubblicazione: (2026)

Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention
di: Guo, Zhenyu, et al.
Pubblicazione: (2025)

Scaling Reasoning without Attention
di: Zhao, Xueliang, et al.
Pubblicazione: (2025)

Cost-Optimal Grouped-Query Attention for Long-Context Modeling
di: Chen, Yingfa, et al.
Pubblicazione: (2025)

SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention
di: Zhu, Qianchao, et al.
Pubblicazione: (2024)

Decomposing Attention To Find Context-Sensitive Neurons
di: Gibson, Alex
Pubblicazione: (2025)

Which Attention Heads Matter for In-Context Learning?
di: Yin, Kayo, et al.
Pubblicazione: (2025)

Depth-Recurrent Attention Mixtures: Giving Latent Reasoning the Attention it Deserves
di: Knupp, Jonas, et al.
Pubblicazione: (2026)

Memorization vs. Reasoning: Updating LLMs with New Knowledge
di: Li, Aochong Oliver, et al.
Pubblicazione: (2025)

Compute Optimal Scaling of Skills: Knowledge vs Reasoning
di: Roberts, Nicholas, et al.
Pubblicazione: (2025)

TIDE: Every Layer Knows the Token Beneath the Context
di: Jaiswal, Ajay, et al.
Pubblicazione: (2026)

MoBA: Mixture of Block Attention for Long-Context LLMs
di: Lu, Enzhe, et al.
Pubblicazione: (2025)

SEAL: Scaling to Emphasize Attention for Long-Context Retrieval
di: Lee, Changhun, et al.
Pubblicazione: (2025)

Longer Context, Deeper Thinking: Uncovering the Role of Long-Context Ability in Reasoning
di: Yang, Wang, et al.
Pubblicazione: (2025)

Attributing Response to Context: A Jensen-Shannon Divergence Driven Mechanistic Study of Context Attribution in Retrieval-Augmented Generation
di: Li, Ruizhe, et al.
Pubblicazione: (2025)

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
di: Hsieh, Cheng-Yu, et al.
Pubblicazione: (2024)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
di: Collins, Liam, et al.
Pubblicazione: (2024)

HiCI: Hierarchical Construction-Integration for Long-Context Attention
di: Zeng, Xiangyu, et al.
Pubblicazione: (2026)

Fortify the Shortest Stave in Attention: Enhancing Context Awareness of Large Language Models for Effective Tool Use
di: Chen, Yuhan, et al.
Pubblicazione: (2023)

What Happened in LLMs Layers when Trained for Fast vs. Slow Thinking: A Gradient Perspective
di: Li, Ming, et al.
Pubblicazione: (2024)

Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions
di: Neo, Clement, et al.
Pubblicazione: (2024)

MeTHanol: Modularized Thinking Language Models with Intermediate Layer Thinking, Decoding and Bootstrapping Reasoning
di: Xi, Ningyuan, et al.
Pubblicazione: (2024)

Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contexts
di: Chen, Yingfa, et al.
Pubblicazione: (2026)

Contextual Drag: How Errors in the Context Affect LLM Reasoning
di: Cheng, Yun, et al.
Pubblicazione: (2026)

Analyzing the Attention Heads for Pronoun Disambiguation in Context-aware Machine Translation Models
di: Mąka, Paweł, et al.
Pubblicazione: (2024)

MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
di: MiniCPM Team, et al.
Pubblicazione: (2026)

Learning Associative Memories with Gradient Descent
di: Cabannes, Vivien, et al.
Pubblicazione: (2024)

FlowRL: Matching Reward Distributions for LLM Reasoning
di: Zhu, Xuekai, et al.
Pubblicazione: (2025)

When Attention Collapses: How Degenerate Layers in LLMs Enable Smaller, Stronger Models
di: Sanyal, Sunny, et al.
Pubblicazione: (2024)

LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
di: Wang, Guangtao, et al.
Pubblicazione: (2025)

Disentangling Logic: The Role of Context in Large Language Model Reasoning Capabilities
di: Hua, Wenyue, et al.
Pubblicazione: (2024)

Dynamic sparsity in tree-structured feed-forward layers at scale
di: Sedghi, Reza, et al.
Pubblicazione: (2026)

DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning
di: Wang, Yaxuan, et al.
Pubblicazione: (2025)

When Thoughts Meet Facts: Reusable Reasoning for Long-Context LMs
di: Jeong, Soyeong, et al.
Pubblicazione: (2025)

Long-Context Reasoning Through Proxy-Based Chain-of-Thought Tuning
di: Li, Miao, et al.
Pubblicazione: (2026)

$π^2$: Structure-Originated Reasoning Data Improves Long-Context Reasoning Ability of Large Language Models
di: Do, Quyet V., et al.
Pubblicazione: (2026)