Cover Image

Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Lu, Yi, Zhao, Wanxu, Zhou, Xin, An, Chenxin, Wang, Chenglong, Li, Shuo, Yang, Yuming, Zhao, Jun, Ji, Tao, Gui, Tao, Zhang, Qi, Huang, Xuanjing
Format:	Preprint
Published:	2025
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Length Generalization of Causal Transformers without Position Encoding
by: Wang, Jie, et al.
Published: (2024)

LongHeads: Multi-Head Attention is Secretly a Long Context Processor
by: Lu, Yi, et al.
Published: (2024)

Unveiling Linguistic Regions in Large Language Models
by: Zhang, Zhihao, et al.
Published: (2024)

Distill Visual Chart Reasoning Ability from LLMs to MLLMs
by: He, Wei, et al.
Published: (2024)

LLaMA Beyond English: An Empirical Study on Language Capability Transfer
by: Zhao, Jun, et al.
Published: (2024)

SafeAligner: Safety Alignment against Jailbreak Attacks via Response Disparity Guidance
by: Huang, Caishuang, et al.
Published: (2024)

Have the VLMs Lost Confidence? A Study of Sycophancy in VLMs
by: Li, Shuo, et al.
Published: (2024)

Mitigating Object Hallucinations in MLLMs via Multi-Frequency Perturbations
by: Li, Shuo, et al.
Published: (2025)

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
by: Zheng, Chuanyang, et al.
Published: (2024)

Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models
by: Zhou, Xin, et al.
Published: (2025)

LongAgent: Scaling Language Models to 128k Context through Multi-Agent Collaboration
by: Zhao, Jun, et al.
Published: (2024)

Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Models
by: He, Wei, et al.
Published: (2024)

AgentPRM: Process Reward Models for LLM Agents via Step-Wise Promise and Progress
by: Xi, Zhiheng, et al.
Published: (2025)

An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention
by: Ghosh, Madhusudan, et al.
Published: (2026)

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding
by: Zhao, Liang, et al.
Published: (2023)

60 Data Points are Sufficient to Fine-Tune LLMs for Question-Answering
by: Ye, Junjie, et al.
Published: (2024)

Analyzing the Effects of Supervised Fine-Tuning on Model Knowledge from Token and Parameter Levels
by: Ye, Junjie, et al.
Published: (2025)

Advancing Translation Preference Modeling with RLHF: A Step Towards Cost-Effective Solution
by: Xu, Nuo, et al.
Published: (2024)

Domain Generalization via Causal Adjustment for Cross-Domain Sentiment Analysis
by: Wang, Siyin, et al.
Published: (2024)

Subspace Defense: Discarding Adversarial Perturbations by Learning a Subspace for Clean Signals
by: Zheng, Rui, et al.
Published: (2024)

The Role of Entropy in Visual Grounding: Analysis and Optimization
by: Li, Shuo, et al.
Published: (2025)

Scaling Laws of RoPE-based Extrapolation
by: Liu, Xiaoran, et al.
Published: (2023)

Measuring Data Diversity for Instruction Tuning: A Systematic Analysis and A Reliable Metric
by: Yang, Yuming, et al.
Published: (2025)

RIFLEx: A Free Lunch for Length Extrapolation in Video Diffusion Transformers
by: Zhao, Min, et al.
Published: (2025)

Are Large Language Models Good Prompt Optimizers?
by: Ma, Ruotian, et al.
Published: (2024)

RoPETR: Improving Temporal Camera-Only 3D Detection by Integrating Enhanced Rotary Position Embedding
by: Ji, Hang, et al.
Published: (2025)

CCTU: A Benchmark for Tool Use under Complex Constraints
by: Ye, Junjie, et al.
Published: (2026)

Large-Momentum Effective Theory's Asymptotic Extrapolation vs the Inverse Problem
by: Chen, Jiunn-Wei, et al.
Published: (2025)

LLM-DA: Data Augmentation via Large Language Models for Few-Shot Named Entity Recognition
by: Ye, Junjie, et al.
Published: (2024)

Disentangling Linguistic Features with Dimension-Wise Analysis of Vector Embeddings
by: Karwa, Saniya, et al.
Published: (2025)

Why Does the Effective Context Length of LLMs Fall Short?
by: An, Chenxin, et al.
Published: (2024)

Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation
by: Lee, Philip Heejun
Published: (2025)

MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation
by: Gao, Weiguo
Published: (2024)

HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position
by: Zhang, Kechi, et al.
Published: (2024)

Why Reinforcement Fine-Tuning Enables MLLMs Preserve Prior Knowledge Better: A Data Perspective
by: Zhang, Zhihao, et al.
Published: (2025)

CLEX: Continuous Length Extrapolation for Large Language Models
by: Chen, Guanzheng, et al.
Published: (2023)

Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms
by: Li, Kewei, et al.
Published: (2025)

Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs
by: Ma, Xin, et al.
Published: (2024)

Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
by: Hua, Ermo, et al.
Published: (2024)

Improving RL Exploration for LLM Reasoning through Retrospective Replay
by: Dou, Shihan, et al.
Published: (2025)