Copertina

Length Extrapolation of Transformers: A Survey from the Perspective of Positional Encoding

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Zhao, Liang, Feng, Xiachong, Feng, Xiaocheng, Zhong, Weihong, Xu, Dongliang, Yang, Qing, Liu, Hongtao, Qin, Bing, Liu, Ting
Natura:	Preprint
Pubblicazione:	2023
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Extending Context Window of Large Language Models from a Distributional Perspective
di: Wu, Yingsheng, et al.
Pubblicazione: (2024)

GlobeSumm: A Challenging Benchmark Towards Unifying Multi-lingual, Cross-lingual and Multi-document News Summarization
di: Ye, Yangfan, et al.
Pubblicazione: (2024)

Advancing Large Language Model Attribution through Self-Improving
di: Huang, Lei, et al.
Pubblicazione: (2024)

Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis
di: Ma, Weitao, et al.
Pubblicazione: (2024)

Adapter-based Selective Knowledge Distillation for Federated Multi-domain Meeting Summarization
di: Feng, Xiachong, et al.
Pubblicazione: (2023)

Beyond Fixed Length: Bucket Pre-training is All You Need
di: Yang, Qing, et al.
Pubblicazione: (2024)

Length Controlled Generation for Black-box LLMs
di: Gu, Yuxuan, et al.
Pubblicazione: (2024)

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
di: He, Zhenyu, et al.
Pubblicazione: (2024)

DAPE: Data-Adaptive Positional Encoding for Length Extrapolation
di: Zheng, Chuanyang, et al.
Pubblicazione: (2024)

A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
di: Huang, Lei, et al.
Pubblicazione: (2023)

Can Large Language Models Simulate Human Cognition Beyond Behavioral Imitation?
di: Gu, Yuxuan, et al.
Pubblicazione: (2026)

Learning Fine-Grained Grounded Citations for Attributed Large Language Models
di: Huang, Lei, et al.
Pubblicazione: (2024)

CultureForest: Understanding and Evaluating Cultural Norm Grounded Reasoning in LLMs
di: Ye, Yangfan, et al.
Pubblicazione: (2026)

Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective
di: Yu, Xinmiao, et al.
Pubblicazione: (2024)

From Hypothesis to Publication: A Comprehensive Survey of AI-Driven Research Support Systems
di: Zhou, Zekun, et al.
Pubblicazione: (2025)

Improving Contextual Faithfulness of Large Language Models via Retrieval Heads-Induced Optimization
di: Huang, Lei, et al.
Pubblicazione: (2025)

PERSONA: Dynamic and Compositional Inference-Time Personality Control via Activation Vector Algebra
di: Feng, Xiachong, et al.
Pubblicazione: (2026)

Meaningful Learning: Enhancing Abstract Reasoning in Large Language Models via Generic Fact Guidance
di: Xiong, Kai, et al.
Pubblicazione: (2024)

One for All: Update Parameterized Knowledge Across Multiple Models
di: Ma, Weitao, et al.
Pubblicazione: (2025)

The Granularity Axis: A Micro-to-Macro Latent Direction for Social Roles in Language Models
di: Qin, Chonghan, et al.
Pubblicazione: (2026)

FroM: Frobenius Norm-Based Data-Free Adaptive Model Merging
di: Li, Zijian, et al.
Pubblicazione: (2025)

Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models
di: Zhong, Weihong, et al.
Pubblicazione: (2024)

Fine-Mem: Fine-Grained Feedback Alignment for Long-Horizon Memory Management
di: Ma, Weitao, et al.
Pubblicazione: (2026)

How does Architecture Influence the Base Capabilities of Pre-trained Language Models? A Case Study Based on FFN-Wider and MoE Transformers
di: Lu, Xin, et al.
Pubblicazione: (2024)

CLEX: Continuous Length Extrapolation for Large Language Models
di: Chen, Guanzheng, et al.
Pubblicazione: (2023)

Length Generalization of Causal Transformers without Position Encoding
di: Wang, Jie, et al.
Pubblicazione: (2024)

ExPe: Exact Positional Encodings for Generative Transformer Models with Extrapolating Capabilities
di: Datseris, Aleksis, et al.
Pubblicazione: (2025)

Ensemble Learning for Heterogeneous Large Language Models with Deep Parallel Collaboration
di: Huang, Yichong, et al.
Pubblicazione: (2024)

LangGPS: Language Separability Guided Data Pre-Selection for Joint Multilingual Instruction Tuning
di: Ye, Yangfan, et al.
Pubblicazione: (2025)

Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation
di: Bianchessi, Arthur S., et al.
Pubblicazione: (2025)

Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation
di: Lu, Yi, et al.
Pubblicazione: (2025)

Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigation
di: Li, Qiming, et al.
Pubblicazione: (2025)

Aligning Translation-Specific Understanding to General Understanding in Large Language Models
di: Huang, Yichong, et al.
Pubblicazione: (2024)

From Interpolation to Extrapolation: Complete Length Generalization for Arithmetic Transformers
di: Duan, Shaoxiong, et al.
Pubblicazione: (2023)

x1: Learning to Think Adaptively Across Languages and Cultures
di: Ye, Yangfan, et al.
Pubblicazione: (2026)

Extrapolation by Association: Length Generalization Transfer in Transformers
di: Cai, Ziyang, et al.
Pubblicazione: (2025)

Trends in Integration of Knowledge and Large Language Models: A Survey and Taxonomy of Methods, Benchmarks, and Applications
di: Feng, Zhangyin, et al.
Pubblicazione: (2023)

Improving the Downstream Performance of Mixture-of-Experts Transformers via Weak Vanilla Transformers
di: Lu, Xin, et al.
Pubblicazione: (2024)

Position as Probability: Self-Supervised Transformers that Think Past Their Training for Length Extrapolation
di: Lee, Philip Heejun
Pubblicazione: (2025)

Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers
di: Hallam, Mohamed Amine, et al.
Pubblicazione: (2026)