Image de couverture de livre

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Zheng, Chuanyang, Gao, Yihang, Shi, Han, Huang, Minbin, Li, Jingyao, Xiong, Jing, Ren, Xiaozhe, Ng, Michael, Jiang, Xin, Li, Zhenguo, Li, Yu
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

DAPE V2: Process Attention Score as Feature Map for Length Extrapolation
par: Zheng, Chuanyang, et autres
Publié: (2024)

Self-Adjust Softmax
par: Zheng, Chuanyang, et autres
Publié: (2025)

Answer-Consistent Chain-of-thought Reinforcement Learning For Multi-modal Large Langauge Models
par: Huang, Minbin, et autres
Publié: (2025)

Efficient Multi-modal Large Language Models via Visual Token Grouping
par: Huang, Minbin, et autres
Publié: (2024)

SepLLM: Accelerate Large Language Models by Compressing One Segment into One Separator
par: Chen, Guoxuan, et autres
Publié: (2024)

AlgoFormer: An Efficient Transformer Framework with Algorithmic Structures
par: Gao, Yihang, et autres
Publié: (2024)

MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation
par: Gao, Weiguo
Publié: (2024)

ParallelComp: Parallel Long-Context Compressor for Length Extrapolation
par: Xiong, Jing, et autres
Publié: (2025)

QuickLLaMA: Query-aware Inference Acceleration for Large Language Models
par: Li, Jingyao, et autres
Publié: (2024)

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding
par: Zhao, Liang, et autres
Publié: (2023)

Mesa-Extrapolation: A Weave Position Encoding Method for Enhanced Extrapolation in LLMs
par: Ma, Xin, et autres
Publié: (2024)

Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
par: Lu, Yi, et autres
Publié: (2025)

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
par: He, Zhenyu, et autres
Publié: (2024)

Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
par: Bianchessi, Arthur S., et autres
Publié: (2025)

UniPool: A Globally Shared Expert Pool for Mixture-of-Experts
par: Huang, Minbin, et autres
Publié: (2026)

Progressive-Hint Prompting Improves Reasoning in Large Language Models
par: Zheng, Chuanyang, et autres
Publié: (2023)

Logits-Based Finetuning
par: Li, Jingyao, et autres
Publié: (2025)

HiRoPE: Length Extrapolation for Code Models Using Hierarchical Position
par: Zhang, Kechi, et autres
Publié: (2024)

CLEX: Continuous Length Extrapolation for Large Language Models
par: Chen, Guanzheng, et autres
Publié: (2023)

LaMPE: Length-aware Multi-grained Positional Encoding for Adaptive Long-context Scaling Without Training
par: Zhang, Sikui, et autres
Publié: (2025)

RoboCoder: Robotic Learning from Basic Skills to General Tasks with Large Language Models
par: Li, Jingyao, et autres
Publié: (2024)

Extending the Width of Short Aperture Data by Data-Driven Extrapolation
par: Schuster, Gerard, et autres
Publié: (2025)

A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)
par: Li, Yan, et autres
Publié: (2025)

SEDGE: Structural Extrapolated Data Generation
par: Zhang, Kun, et autres
Publié: (2026)

Getting More Juice Out of Your Data: Hard Pair Refinement Enhances Visual-Language Models Without Extra Data
par: Wang, Haonan, et autres
Publié: (2023)

Positional Encoding Field
par: Bai, Yunpeng, et autres
Publié: (2025)

LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
par: Zhang, Shen, et autres
Publié: (2025)

Toward Relative Positional Encoding in Spiking Transformers
par: Lv, Changze, et autres
Publié: (2025)

Length Generalization of Causal Transformers without Position Encoding
par: Wang, Jie, et autres
Publié: (2024)

Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms
par: Li, Kewei, et autres
Publié: (2025)

The Linear Attention Resurrection in Vision Transformer
par: Zheng, Chuanyang
Publié: (2025)

iFormer: Integrating ConvNet and Transformer for Mobile Application
par: Zheng, Chuanyang
Publié: (2025)

Understanding the Mixture-of-Experts with Nadaraya-Watson Kernel
par: Zheng, Chuanyang, et autres
Publié: (2025)

Lyra: Orchestrating Dual Correction in Automated Theorem Proving
par: Zheng, Chuanyang, et autres
Publié: (2023)

DESTEIN: Navigating Detoxification of Language Models via Universal Steering Pairs and Head-wise Activation Fusion
par: Li, Yu, et autres
Publié: (2024)

Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation
par: Lee, Philip Heejun
Publié: (2025)

ExPe: Exact Positional Encodings for Generative Transformer Models with Extrapolating Capabilities
par: Datseris, Aleksis, et autres
Publié: (2025)

Extrapolation Merging: Keep Improving With Extrapolation and Merging
par: Lin, Yiguan, et autres
Publié: (2025)

From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers
par: Duan, Shaoxiong, et autres
Publié: (2023)

Extracellular vesicles—Potential link between periodontal disease and diabetic complications
par: Shengyuan Huang, et autres
Publié: (2024)