Cover Image

Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Hua, Ermo, Jiang, Che, Lv, Xingtai, Zhang, Kaiyan, Sun, Youbang, Fan, Yuchen, Zhu, Xuekai, Qi, Biqing, Ding, Ning, Zhou, Bowen
Format:	Preprint
Published:	2024
Subjects:	Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
by: Zhang, Kaiyan, et al.
Published: (2024)

Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
by: Hua, Ermo, et al.
Published: (2024)

Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models
by: Lv, Xingtai, et al.
Published: (2025)

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention
by: Lv, Xingtai, et al.
Published: (2024)

CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following
by: Zhang, Kaiyan, et al.
Published: (2024)

How to Synthesize Text Data without Model Collapse?
by: Zhu, Xuekai, et al.
Published: (2024)

UltraMedical: Building Specialized Generalists in Biomedicine
by: Zhang, Kaiyan, et al.
Published: (2024)

Towards a Unified View of Large Language Model Post-Training
by: Lv, Xingtai, et al.
Published: (2025)

Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
by: Long, Xinwei, et al.
Published: (2025)

TTRL: Test-Time Reinforcement Learning
by: Zuo, Yuxin, et al.
Published: (2025)

DePass: Unified Feature Attributing by Simple Decomposed Forward Pass
by: Hong, Xiangyu, et al.
Published: (2025)

PaD: Program-aided Distillation Can Teach Small Models Reasoning Better than Chain-of-thought Fine-tuning
by: Zhu, Xuekai, et al.
Published: (2023)

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
by: Zuo, Yuxin, et al.
Published: (2025)

Towards Building Specialized Generalist AI with System 1 and System 2 Fusion
by: Zhang, Kaiyan, et al.
Published: (2024)

Automating Exploratory Proteomics Research via Language Models
by: Ding, Ning, et al.
Published: (2024)

Automating Exploratory Multiomics Research via Language Models
by: Qu, Shang, et al.
Published: (2025)

FlowRL: Matching Reward Distributions for LLM Reasoning
by: Zhu, Xuekai, et al.
Published: (2025)

Post-Trained MoE Can Skip Half Experts via Self-Distillation
by: Lv, Xingtai, et al.
Published: (2026)

Large Language Models as Biomedical Hypothesis Generators: A Comprehensive Evaluation
by: Qi, Biqing, et al.
Published: (2024)

How Far Can Unsupervised RLVR Scale LLM Training?
by: He, Bingxiang, et al.
Published: (2026)

Nirvana: A Specialized Generalist Model With Task-Aware Memory Mechanism
by: Jiang, Yuhua, et al.
Published: (2025)

A Survey of Reinforcement Learning for Large Reasoning Models
by: Zhang, Kaiyan, et al.
Published: (2025)

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
by: Qi, Biqing, et al.
Published: (2024)

SSRL: Self-Search Reinforcement Learning
by: Fan, Yuchen, et al.
Published: (2025)

MSI-Agent: Incorporating Multi-Scale Insight into Embodied Agents for Superior Planning and Decision-Making
by: Fu, Dayuan, et al.
Published: (2024)

SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
by: Li, Haozhan, et al.
Published: (2025)

Evolution of Thought: Diverse and High-Quality Reasoning via Multi-Objective Optimization
by: Qi, Biqing, et al.
Published: (2024)

SMR: State Memory Replay for Long Sequence Modeling
by: Qi, Biqing, et al.
Published: (2024)

Correlation-Aware Select and Merge Attention for Efficient Fine-Tuning and Context Length Extension
by: Wang, Ning, et al.
Published: (2024)

Enhancing Adversarial Transferability via Information Bottleneck Constraints
by: Qi, Biqing, et al.
Published: (2024)

On the token distance modeling ability of higher RoPE attention dimension
by: Hong, Xiangyu, et al.
Published: (2024)

Can 1B LLM Surpass 405B LLM? Rethinking Compute-Optimal Test-Time Scaling
by: Liu, Runze, et al.
Published: (2025)

Self-Reflective Reinforcement Learning for Diffusion-based Image Reasoning Generation
by: Pan, Jiadong, et al.
Published: (2025)

LFQA-E: Carefully Benchmarking Long-form QA Evaluation
by: Fan, Yuchen, et al.
Published: (2024)

An Evaluation of Context Length Extrapolation in Long Code via Positional Embeddings and Efficient Attention
by: Ghosh, Madhusudan, et al.
Published: (2026)

Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
by: Ding, Ning, et al.
Published: (2024)

Safe-SD: Safe and Traceable Stable Diffusion with Text Prompt Trigger for Invisible Generative Watermarking
by: Ma, Zhiyuan, et al.
Published: (2024)

Neural Residual Diffusion Models for Deep Scalable Vision Generation
by: Ma, Zhiyuan, et al.
Published: (2024)

Leveraging Multi-lingual Positive Instances in Contrastive Learning to Improve Sentence Embedding
by: Zhao, Kaiyan, et al.
Published: (2023)

Critical Data Size of Language Models from a Grokking Perspective
by: Zhu, Xuekai, et al.
Published: (2024)