Image de couverture de livre

Empirical Study on Updating Key-Value Memories in Transformer Feed-forward Layers

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Qiu, Zihan, Huang, Zeyu, Huang, Youcheng, Fu, Jie
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

A Closer Look into Mixture-of-Experts in Large Language Models
par: Lo, Ka Man, et autres
Publié: (2024)

Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
par: Du, Wenyu, et autres
Publié: (2024)

Layerwise Recurrent Router for Mixture-of-Experts
par: Qiu, Zihan, et autres
Publié: (2024)

Distributional Associations vs In-Context Reasoning: A Study of Feed-forward and Attention Layers
par: Chen, Lei, et autres
Publié: (2024)

F-MALLOC: Feed-forward Memory Allocation for Continual Learning in Neural Machine Translation
par: Wu, Junhong, et autres
Publié: (2024)

Post-hoc Reward Calibration: A Case Study on Length Bias
par: Huang, Zeyu, et autres
Publié: (2024)

Reducing Transformer Key-Value Cache Size with Cross-Layer Attention
par: Brandon, William, et autres
Publié: (2024)

Unlocking Continual Learning Abilities in Language Models
par: Du, Wenyu, et autres
Publié: (2024)

Steering Information Utility in Key-Value Memory for Language Model Post-Training
par: Deng, Chunyuan, et autres
Publié: (2025)

Trellis: Learning to Compress Key-Value Memory in Attention Models
par: Karami, Mahdi, et autres
Publié: (2025)

A Controllable Examination for Long-Context Language Models
par: Yang, Yijun, et autres
Publié: (2025)

Cross-model Transferability among Large Language Models on the Platonic Representations of Concepts
par: Huang, Youcheng, et autres
Publié: (2025)

Key-Value Means: Transformers with Expandable Block-Recurrent Compressed Memory
par: Goldstein, Daniel, et autres
Publié: (2026)

See the Unseen: Better Context-Consistent Knowledge-Editing by Noises
par: Huang, Youcheng, et autres
Publié: (2024)

MemoryFormer: Minimize Transformer Computation by Removing Fully-Connected Layers
par: Ding, Ning, et autres
Publié: (2024)

Evidence Absence Is Not Evidence Insufficiency: Diagnosing NEI Construction Artifacts in Fact Verification
par: Qiu, Jingxi, et autres
Publié: (2026)

Assessing Adversarial Robustness of Large Language Models: An Empirical Study
par: Yang, Zeyu, et autres
Publié: (2024)

SpeCache: Speculative Key-Value Caching for Efficient Generation of LLMs
par: Jie, Shibo, et autres
Publié: (2025)

Lying with Truths: Open-Channel Multi-Agent Collusion for Belief Manipulation via Generative Montage
par: Hu, Jinwei, et autres
Publié: (2026)

Strong Memory, Weak Control: An Empirical Study of Executive Functioning in LLMs
par: de Langis, Karin, et autres
Publié: (2025)

On the Transformations across Reward Model, Parameter Update, and In-Context Prompt
par: Cai, Deng, et autres
Publié: (2024)

Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
par: Feng, Duanyu, et autres
Publié: (2024)

Rethinking Attention: Exploring Shallow Feed-Forward Neural Networks as an Alternative to Attention Layers in Transformers
par: Bozic, Vukasin, et autres
Publié: (2023)

Identifying Speaker Information in Feed-Forward Layers of Self-Supervised Speech Transformers
par: Lin, Tzu-Quan, et autres
Publié: (2025)

CodeUpdateArena: Benchmarking Knowledge Editing on API Updates
par: Liu, Zeyu Leo, et autres
Publié: (2024)

What Training Data Teaches RL Memory Agents: An Empirical Study of Curriculum Effects in Memory-Augmented QA
par: He, Xinjie, et autres
Publié: (2026)

DictLLM: Harnessing Key-Value Data Structures with Large Language Models for Enhanced Medical Diagnostics
par: Guo, YiQiu, et autres
Publié: (2024)

Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
par: Qiu, Zihan, et autres
Publié: (2025)

MoRA: High-Rank Updating for Parameter-Efficient Fine-Tuning
par: Jiang, Ting, et autres
Publié: (2024)

Spatial ModernBERT: Spatial-Aware Transformer for Table and Key-Value Extraction in Financial Documents at Scale
par: Javis AI Team, et autres
Publié: (2025)

"You Are Rejected!": An Empirical Study of Large Language Models Taking Hiring Evaluations
par: Fu, Dingjie, et autres
Publié: (2025)

Most Transformer Modifications Still Do Not Transfer at 1-3B: A 2020-2026 Update to Narang et al. (2021) with Downstream Evaluation and a Noise Floor
par: Zhao, Yang, et autres
Publié: (2026)

A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
par: Qiu, Zihan, et autres
Publié: (2026)

Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
par: Huang, Zeyu, et autres
Publié: (2025)

Signatures of human-like processing in Transformer forward passes
par: Hu, Jennifer, et autres
Publié: (2025)

Beyond Solving Math Quiz: Evaluating the Ability of Large Reasoning Models to Ask for Information
par: Huang, Youcheng, et autres
Publié: (2025)

Responsible Agentic AI Requires Explicit Provenance
par: Hu, Jinwei, et autres
Publié: (2026)

Think Before You Act: Decision Transformers with Working Memory
par: Kang, Jikun, et autres
Publié: (2023)

Unlocking Emergent Modularity in Large Language Models
par: Qiu, Zihan, et autres
Publié: (2023)

LLM Braces: Straightening Out LLM Predictions with Relevant Sub-Updates
par: Shen, Ying, et autres
Publié: (2025)