Image de couverture de livre

Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Ben-Artzy, Amit, Schwartz, Roy
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

SpeLLM: Character-Level Multi-Head Decoding
par: Ben-Artzy, Amit, et autres
Publié: (2025)

Attend to Evidence: Evidence-Anchored Spatial Attention Supervision for Multimodal RLVR
par: Hu, Ruina, et autres
Publié: (2026)

Should We Attend More or Less? Modulating Attention for Fairness
par: Zayed, Abdelrahman, et autres
Publié: (2023)

Beyond Performance: Quantifying and Mitigating Label Bias in LLMs
par: Reif, Yuval, et autres
Publié: (2024)

Consolidating Rewarded Perturbations for LLM Post-Training
par: Zhang, Zheyu, et autres
Publié: (2026)

"It's how you do things that matters": Attending to Process to Better Serve Indigenous Communities with Language Technologies
par: Cooper, Ned, et autres
Publié: (2024)

Model Tells Itself Where to Attend: Faithfulness Meets Automatic Attention Steering
par: Zhang, Qingru, et autres
Publié: (2024)

Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMs
par: Zhang, Qingru, et autres
Publié: (2023)

Don't Overthink it. Preferring Shorter Thinking Chains for Improved LLM Reasoning
par: Hassid, Michael, et autres
Publié: (2025)

Clean First, Align Later: Benchmarking Preference Data Cleaning for Reliable LLM Alignment
par: Yeh, Samuel, et autres
Publié: (2025)

XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
par: Monteiro, João, et autres
Publié: (2024)

Compressible Softmax-Attended Language under Incompressible Attention
par: Lee, Wonsuk
Publié: (2026)

CLAA: Cross-Layer Attention Aggregation for Accelerating LLM Prefill
par: McDanel, Bradley, et autres
Publié: (2026)

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
par: Ma, Da, et autres
Publié: (2024)

Investigating Layer Importance in Large Language Models
par: Zhang, Yang, et autres
Publié: (2024)

Vocab Diet: Reshaping the Vocabulary of LLMs via Vector Arithmetic
par: Reif, Yuval, et autres
Publié: (2025)

Love First, Know Later: Persona-Based Romantic Compatibility Through LLM Text World Engines
par: Shang, Haoyang, et autres
Publié: (2025)

Learning When Not to Attend Globally
par: Luo, Xuan, et autres
Publié: (2025)

Inferring Functionality of Attention Heads from their Parameters
par: Elhelo, Amit, et autres
Publié: (2024)

Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
par: Körner, Felicia, et autres
Publié: (2026)

Reversed Attention: On The Gradient Descent Of Attention Layers In GPT
par: Katz, Shahar, et autres
Publié: (2024)

Compute First: Read and Write Later.
par: Monahan, Brian D., et autres
Publié: (1982)

HTAM: Hierarchical Transition-Attended Memory for Operator Optimization
par: Zhang, Yining, et autres
Publié: (2026)

Hi-ZFO: Hierarchical Zeroth- and First-Order LLM Fine-Tuning via Importance-Guided Tensor Selection
par: Jin, Feihu, et autres
Publié: (2026)

Attamba: Attending To Multi-Token States
par: Akhauri, Yash, et autres
Publié: (2024)

Why Attend to Everything? Focus is the Key
par: Yao, Hengshuai, et autres
Publié: (2026)

Algorithm for Automatic Legislative Text Consolidation
par: Etcheverry, Matias, et autres
Publié: (2025)

Importance-Aware Data Selection for Efficient LLM Instruction Tuning
par: Jiang, Tingyu, et autres
Publié: (2025)

First-Step Advantage: Importance of Starting Right in Multi-Step Math Reasoning
par: Jain, Kushal, et autres
Publié: (2023)

Non-verbal information in spontaneous speech -- towards a new framework of analysis
par: Biron, Tirza, et autres
Publié: (2024)

MTMT: Consolidating Multiple Thinking Modes to Form a Thought Tree for Strengthening LLM
par: Li, Changcheng, et autres
Publié: (2024)

Selective Attention: Enhancing Transformer through Principled Context Control
par: Zhang, Xuechen, et autres
Publié: (2024)

On Pruning State-Space LLMs
par: Ghattas, Tamer, et autres
Publié: (2025)

The Importance of Directional Feedback for LLM-based Optimizers
par: Nie, Allen, et autres
Publié: (2024)

Towards Objective and Unbiased Decision Assessments with LLM-Enhanced Hierarchical Attention Networks
par: Liu, Junhua, et autres
Publié: (2024)

AVSS: Layer Importance Evaluation in Large Language Models via Activation Variance-Sparsity Analysis
par: Song, Zichen, et autres
Publié: (2024)

On Importance of Layer Pruning for Smaller BERT Models and Low Resource Languages
par: Shirke, Mayur, et autres
Publié: (2025)

PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression
par: Chen, Lizhe, et autres
Publié: (2025)

Attention Is Not All You Need: The Importance of Feedforward Networks in Transformer Models
par: Gerber, Isaac
Publié: (2025)

Ask Now, Use Later: Benchmarking the Proactivity Gap in Long-Lived LLM Agents
par: Wu, Bin, et autres
Publié: (2026)