Imagen de Portada

Position Encoding with Random Float Sampling Enhances Length Generalization of Transformers

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Shimizu, Atsushi, Taniguchi, Shohei, Matsuo, Yutaka
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Residual Koopman Spectral Profiling for Predicting and Preventing Transformer Training Instability
por: Kim, Bum Jun, et al.
Publicado: (2026)

$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
por: Takashiro, Shota, et al.
Publicado: (2026)

Fusion Matters: Length-Aware Analysis of Positional-Encoding Fusion in Transformers
por: Hallam, Mohamed Amine, et al.
Publicado: (2026)

MEP: Multiple Kernel Learning Enhancing Relative Positional Encoding Length Extrapolation
por: Gao, Weiguo
Publicado: (2024)

Enhancing Unimodal Latent Representations in Multimodal VAEs through Iterative Amortized Inference
por: Oshima, Yuta, et al.
Publicado: (2024)

On the Geometry of Positional Encodings in Transformers
por: Cirrincione, Giansalvo
Publicado: (2026)

Improving Transformers using Faithful Positional Encoding
por: Idé, Tsuyoshi, et al.
Publicado: (2024)

Comparing Graph Transformers via Positional Encodings
por: Black, Mitchell, et al.
Publicado: (2024)

A Causal Inference Approach for Quantifying Research Impact
por: Ochiai, Keiichi, et al.
Publicado: (2025)

Graph Transformers without Positional Encodings
por: Garg, Ayush
Publicado: (2024)

ADOPT: Modified Adam Can Converge with Any $β_2$ with the Optimal Rate
por: Taniguchi, Shohei, et al.
Publicado: (2024)

Towards Empirical Interpretation of Internal Circuits and Properties in Grokked Transformers on Modular Polynomials
por: Furuta, Hiroki, et al.
Publicado: (2024)

Size Transferability of Graph Transformers with Convolutional Positional Encodings
por: Porras-Valenzuela, Javier, et al.
Publicado: (2026)

Vocabulary In-Context Learning in Transformers: Benefits of Positional Encoding
por: Ma, Qian, et al.
Publicado: (2025)

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers
por: Choromanski, Krzysztof Marcin, et al.
Publicado: (2023)

Improved Active Learning via Dependent Leverage Score Sampling
por: Shimizu, Atsushi, et al.
Publicado: (2023)

Use of Prior Knowledge to Discover Causal Additive Models with Unobserved Variables and its Application to Time Series Data
por: Maeda, Takashi Nicholas, et al.
Publicado: (2024)

Bridging Lottery Ticket and Grokking: Understanding Grokking from Inner Structure of Networks
por: Minegishi, Gouki, et al.
Publicado: (2023)

Benchmarking Positional Encodings for GNNs and Graph Transformers
por: Grötschla, Florian, et al.
Publicado: (2024)

Looped Transformers for Length Generalization
por: Fan, Ying, et al.
Publicado: (2024)

Dynamic Graph Transformer with Correlated Spatial-Temporal Positional Encoding
por: Wang, Zhe, et al.
Publicado: (2024)

Tab-PET: Graph-Based Positional Encodings for Tabular Transformers
por: Leng, Yunze, et al.
Publicado: (2025)

Counterfactual Explanations of Black-box Machine Learning Models using Causal Discovery with Applications to Credit Rating
por: Takahashi, Daisuke, et al.
Publicado: (2024)

What Improves the Generalization of Graph Transformers? A Theoretical Dive into the Self-attention and Positional Encoding
por: Li, Hongkang, et al.
Publicado: (2024)

On the Expressive Power of Floating-Point Transformers
por: Park, Sejun, et al.
Publicado: (2026)

Graph Transformer with Disease Subgraph Positional Encoding for Improved Comorbidity Prediction
por: Qin, Xihan, et al.
Publicado: (2025)

Positional Encoding in Transformer-Based Time Series Models: A Survey
por: Irani, Habib, et al.
Publicado: (2025)

PEPS: Positional Encoding Projected Sampling -- Extended
por: Perez, Guillaume, et al.
Publicado: (2026)

Quantitative Bounds for Length Generalization in Transformers
por: Izzo, Zachary, et al.
Publicado: (2025)

Bisimulation metric for Model Predictive Control
por: Shimizu, Yutaka, et al.
Publicado: (2024)

Density Ratio-based Causal Discovery from Bivariate Continuous-Discrete Data
por: Maeda, Takashi Nicholas, et al.
Publicado: (2025)

Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure
por: Cho, Hanseul, et al.
Publicado: (2024)

Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation
por: He, Zhenyu, et al.
Publicado: (2024)

On the Limitations and Capabilities of Position Embeddings for Length Generalization
por: Chen, Yang, et al.
Publicado: (2025)

Retrieve-Augmented Generation for Speeding up Diffusion Policy without Additional Training
por: Odonchimed, Sodtavilan, et al.
Publicado: (2025)

Safe Transformer: An Explicit Safety Bit For Interpretable And Controllable Alignment
por: Feng, Jingyuan, et al.
Publicado: (2026)

Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks
por: Sabbaghi, Mahdi, et al.
Publicado: (2024)

Causal Additive Models with Unobserved Causal Paths and Backdoor Paths
por: Pham, Thong, et al.
Publicado: (2025)

SSM Meets Video Diffusion Models: Efficient Long-Term Video Generation with Structured State Spaces
por: Oshima, Yuta, et al.
Publicado: (2024)

RelFlexformer: Efficient Attention 3D-Transformers for Integrable Relative Positional Encodings
por: Kim, Byeongchan, et al.
Publicado: (2026)