Cover Image

Softplus Attention with Re-weighting Boosts Length Extrapolation in Large Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Gao, Bo, Spratling, Michael W., Gionfrida, Letizia
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Instruction Following by Principled Boosting Attention of Large Language Models
by: Guardieiro, Vitoria, et al.
Published: (2025)

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
by: He, Zhenyu, et al.
Published: (2024)

One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models
by: Li, Lin, et al.
Published: (2024)

Boosting Large Language Models with Mask Fine-Tuning
by: Zhang, Mingyuan, et al.
Published: (2025)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
by: Peng, Runyu, et al.
Published: (2026)

Model Extrapolation Expedites Alignment
by: Zheng, Chujie, et al.
Published: (2024)

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
by: Leng, Jiaqi, et al.
Published: (2025)

Length-MAX Tokenizer for Language Models
by: Dong, Dong, et al.
Published: (2025)

BoostStep: Boosting mathematical capability of Large Language Models via improved single-step reasoning
by: Zhang, Beichen, et al.
Published: (2025)

ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attacks
by: Yu, Xiaodong, et al.
Published: (2023)

Boosting of Thoughts: Trial-and-Error Problem Solving with Large Language Models
by: Chen, Sijia, et al.
Published: (2024)

Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation
by: Lee, Philip Heejun
Published: (2025)

TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selection
by: Wu, Wei, et al.
Published: (2024)

On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
by: Terzić, Aleksandar, et al.
Published: (2024)

Sliding Window Attention Training for Efficient Large Language Models
by: Fu, Zichuan, et al.
Published: (2025)

Position Engineering: Boosting Large Language Models through Positional Information Manipulation
by: He, Zhiyuan, et al.
Published: (2024)

SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengths
by: Huang, Kaixuan, et al.
Published: (2024)

RePo: Language Models with Context Re-Positioning
by: Li, Huayang, et al.
Published: (2025)

ReFusion: A Diffusion Large Language Model with Parallel Autoregressive Decoding
by: Li, Jia-Nan, et al.
Published: (2025)

Re-Tuning: Overcoming the Compositionality Limits of Large Language Models with Recursive Tuning
by: Pasewark, Eric, et al.
Published: (2024)

RePrompt: Planning by Automatic Prompt Engineering for Large Language Models Agents
by: Chen, Weizhe, et al.
Published: (2024)

Large Language Models Badly Generalize across Option Length, Problem Types, and Irrelevant Noun Replacements
by: Zhao, Guangxiang, et al.
Published: (2025)

On the Optimal Reasoning Length for RL-Trained Language Models
by: Nohara, Daisuke, et al.
Published: (2026)

RLPR: Extrapolating RLVR to General Domains without Verifiers
by: Yu, Tianyu, et al.
Published: (2025)

ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks
by: Zhu, Yinghao, et al.
Published: (2024)

Confidence Regularized Masked Language Modeling using Text Length
by: Ji, Seunghyun, et al.
Published: (2025)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by: Lv, Xingtai, et al.
Published: (2024)

Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Results
by: Santilli, Andrea, et al.
Published: (2025)

Effects of Prompt Length on Domain-specific Tasks for Large Language Models
by: Liu, Qibang, et al.
Published: (2025)

E-Sparse: Boosting the Large Language Model Inference through Entropy-based N:M Sparsity
by: Li, Yun, et al.
Published: (2023)

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
by: Guan, Ziyi, et al.
Published: (2024)

ReALM: Reference Resolution As Language Modeling
by: Moniz, Joel Ruben Antony, et al.
Published: (2024)

ReFT: Representation Finetuning for Language Models
by: Wu, Zhengxuan, et al.
Published: (2024)

Self-Generated Critiques Boost Reward Modeling for Language Models
by: Yu, Yue, et al.
Published: (2024)

Boosting Protein Language Models with Negative Sample Mining
by: Xu, Yaoyao, et al.
Published: (2024)

Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models
by: Duan, Jinhao, et al.
Published: (2023)

Context-Aware Initialization for Reducing Generative Path Length in Diffusion Language Models
by: Miao, Tongyuan, et al.
Published: (2025)

FCoReBench: Can Large Language Models Solve Challenging First-Order Combinatorial Reasoning Problems?
by: Mittal, Chinmay, et al.
Published: (2024)

Rethinking Interpretability in the Era of Large Language Models
by: Singh, Chandan, et al.
Published: (2024)

Multiscale Byte Language Models -- A Hierarchical Architecture for Causal Million-Length Sequence Modeling
by: Egli, Eric, et al.
Published: (2025)