Image de couverture de livre

Selective Attention Improves Transformer

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Leviathan, Yaniv, Kalman, Matan, Matias, Yossi
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Prompt Repetition Improves Non-Reasoning LLMs
par: Leviathan, Yaniv, et autres
Publié: (2025)

Generative UI: LLMs are Effective UI Generators
par: Leviathan, Yaniv, et autres
Publié: (2026)

Diffusion Models Are Real-Time Game Engines
par: Valevski, Dani, et autres
Publié: (2024)

When Should LLMs Be Less Specific? Selective Abstraction for Reliable Long-Form Text Generation
par: Goren, Shani, et autres
Publié: (2026)

GLU Attention Improve Transformer
par: Wang, Zehao
Publié: (2025)

Sessa: Selective State Space Attention
par: Horbatko, Liubomyr
Publié: (2026)

Intent-based Prompt Calibration: Enhancing prompt optimization with synthetic boundary cases
par: Levi, Elad, et autres
Publié: (2024)

What Matters in Transformers? Not All Attention is Needed
par: He, Shwai, et autres
Publié: (2024)

Forgetting Transformer: Softmax Attention with a Forget Gate
par: Lin, Zhixuan, et autres
Publié: (2025)

The Anxiety of Influence: Bloom Filters in Transformer Attention Heads
par: Balogh, Peter
Publié: (2026)

The Larger the Better? Improved LLM Code-Generation via Budget Reallocation
par: Hassid, Michael, et autres
Publié: (2024)

TxGemma: Efficient and Agentic LLMs for Therapeutics
par: Wang, Eric, et autres
Publié: (2025)

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
par: Leng, Jiaqi, et autres
Publié: (2025)

In-Context Learning with Transformers: Softmax Attention Adapts to Function Lipschitzness
par: Collins, Liam, et autres
Publié: (2024)

RecurFormer: Not All Transformer Heads Need Self-Attention
par: Yan, Ruiqing, et autres
Publié: (2024)

Pooling Attention: Evaluating Pretrained Transformer Embeddings for Deception Classification
par: Mamtani, Sumit, et autres
Publié: (2025)

Integrating Locality-Aware Attention with Transformers for General Geometry PDEs
par: Koh, Minsu, et autres
Publié: (2025)

Taipan: Efficient and Expressive State Space Language Models with Selective Attention
par: Van Nguyen, Chien, et autres
Publié: (2024)

Optimizing Attention with Mirror Descent: Generalized Max-Margin Token Selection
par: Julistiono, Addison Kristanto, et autres
Publié: (2024)

Selective Synchronization Attention
par: Hays, Hasi
Publié: (2026)

From Self-Attention to Markov Models: Unveiling the Dynamics of Generative Transformers
par: Ildiz, M. Emrullah, et autres
Publié: (2024)

Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactions
par: Neo, Clement, et autres
Publié: (2024)

Emergence of Minimal Circuits for Indirect Object Identification in Attention-Only Transformers
par: Adhikari, Rabin
Publié: (2025)

From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformers
par: Su, Jingtong, et autres
Publié: (2025)

Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization
par: Hsieh, Cheng-Yu, et autres
Publié: (2024)

Attention Mechanisms Don't Learn Additive Models: Rethinking Feature Importance for Transformers
par: Leemann, Tobias, et autres
Publié: (2024)

Decoupling Knowledge and Reasoning in Transformers: A Modular Architecture with Generalized Cross-Attention
par: Guo, Zhenyu, et autres
Publié: (2025)

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
par: Tian, Yuandong, et autres
Publié: (2023)

Share Your Attention: Transformer Weight Sharing via Matrix-based Dictionary Learning
par: Zhussip, Magauiya, et autres
Publié: (2025)

A Causal World Model Underlying Next Token Prediction: Exploring GPT in a Controlled Environment
par: Rohekar, Raanan Y., et autres
Publié: (2024)

Less is More: Improving LLM Alignment via Preference Data Selection
par: Deng, Xun, et autres
Publié: (2025)

DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusion
par: Chen, Yilong, et autres
Publié: (2024)

Emergence of Episodic Memory in Transformers: Characterizing Changes in Temporal Structure of Attention Scores During Training
par: Mistry, Deven Mahesh, et autres
Publié: (2025)

Multilingual Instruction Tuning With Just a Pinch of Multilinguality
par: Shaham, Uri, et autres
Publié: (2024)

Unpacking Tokenization: Evaluating Text Compression and its Correlation with Model Performance
par: Goldman, Omer, et autres
Publié: (2024)

MetaGreen: Meta-Learning Inspired Transformer Selection for Green Semantic Communication
par: Mukherjee, Shubhabrata, et autres
Publié: (2024)

CoCA: Fusing Position Embedding with Collinear Constrained Attention in Transformers for Long Context Window Extending
par: Zhu, Shiyi, et autres
Publié: (2023)

Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure
par: Cho, Hanseul, et autres
Publié: (2024)

EcoTransformer: Attention without Multiplication
par: Gao, Xin, et autres
Publié: (2025)

Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
par: Yang, Yu, et autres
Publié: (2024)