Cover Image

Transformers Can Achieve Length Generalization But Not Robustly

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhou, Yongchao, Alon, Uri, Chen, Xinyun, Wang, Xuezhi, Agarwal, Rishabh, Zhou, Denny
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Large Language Models as Tool Makers
by: Cai, Tianle, et al.
Published: (2023)

Large Language Models as Optimizers
by: Yang, Chengrun, et al.
Published: (2023)

On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes
by: Agarwal, Rishabh, et al.
Published: (2023)

Premise Order Matters in Reasoning with Large Language Models
by: Chen, Xinyun, et al.
Published: (2024)

DistillSpec: Improving Speculative Decoding via Knowledge Distillation
by: Zhou, Yongchao, et al.
Published: (2023)

Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models
by: Zheng, Huaixiu Steven, et al.
Published: (2023)

Provable Length Generalization in Sequence Prediction via Spectral Filtering
by: Marsden, Annie, et al.
Published: (2024)

The Role of Sparsity for Length Generalization in Transformers
by: Golowich, Noah, et al.
Published: (2025)

BioCoder: A Benchmark for Bioinformatics Code Generation with Large Language Models
by: Tang, Xiangru, et al.
Published: (2023)

Can LLMs Speak For Diverse People? Tuning LLMs via Debate to Generate Controllable Controversial Statements
by: Li, Ming, et al.
Published: (2024)

Multi-Layer Transformers Gradient Can be Approximated in Almost Linear Time
by: Liang, Yingyu, et al.
Published: (2024)

Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure
by: Cho, Hanseul, et al.
Published: (2024)

WebArena: A Realistic Web Environment for Building Autonomous Agents
by: Zhou, Shuyan, et al.
Published: (2023)

Supernova: Achieving More with Less in Transformer Architectures
by: Tanase, Andrei-Valentin, et al.
Published: (2025)

Can Post-Training Transform LLMs into Causal Reasoners?
by: Chen, Junqi, et al.
Published: (2026)

Transformers Learn to Achieve Second-Order Convergence Rates for In-Context Linear Regression
by: Fu, Deqing, et al.
Published: (2023)

Can an LLM Induce a Graph? Investigating Memory Drift and Context Length
by: Yousuf, Raquib Bin, et al.
Published: (2025)

Asynchronous RLHF: Faster and More Efficient Off-Policy RL for Language Models
by: Noukhovitch, Michael, et al.
Published: (2024)

V-STaR: Training Verifiers for Self-Taught Reasoners
by: Hosseini, Arian, et al.
Published: (2024)

CL4KGE: A Curriculum Learning Method for Knowledge Graph Embedding
by: Liu, Yang, et al.
Published: (2024)

Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models
by: Leng, Jiaqi, et al.
Published: (2025)

SwitchLoRA: Switched Low-Rank Adaptation Can Learn Full-Rank Information
by: Zhou, Kaiye, et al.
Published: (2024)

2-Tier SimCSE: Elevating BERT for Robust Sentence Embeddings
by: Wang, Yumeng, et al.
Published: (2025)

Climbing the Ladder of Reasoning: What LLMs Can-and Still Can't-Solve after SFT?
by: Sun, Yiyou, et al.
Published: (2025)

Process Reward Models That Think
by: Khalifa, Muhammad, et al.
Published: (2025)

When Can Transformers Count to n?
by: Yehudai, Gilad, et al.
Published: (2024)

Learn to Reason Efficiently with Adaptive Length-based Reward Shaping
by: Liu, Wei, et al.
Published: (2025)

Bridging Kolmogorov Complexity and Deep Learning: Asymptotically Optimal Description Length Objectives for Transformers
by: Shaw, Peter, et al.
Published: (2025)

RCStat: A Statistical Framework for using Relative Contextualization in Transformers
by: Mahapatra, Debabrata, et al.
Published: (2025)

An Empirical Study on Context Length for Open-Domain Dialog Generation
by: Shen, Xinyi, et al.
Published: (2024)

On the Robustness of Transformers against Context Hijacking for Linear Classification
by: Li, Tianle, et al.
Published: (2025)

Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Data
by: Wang, Xinyi, et al.
Published: (2024)

Genie: Achieving Human Parity in Content-Grounded Datasets Generation
by: Yehudai, Asaf, et al.
Published: (2024)

C-3PO: Compact Plug-and-Play Proxy Optimization to Achieve Human-like Retrieval-Augmented Generation
by: Chen, Guoxin, et al.
Published: (2025)

Towards Compute-Optimal Many-Shot In-Context Learning
by: Golchin, Shahriar, et al.
Published: (2025)

Can AI-Generated Text be Reliably Detected?
by: Sadasivan, Vinu Sankar, et al.
Published: (2023)

Unraveling LoRA Interference: Orthogonal Subspaces for Robust Model Merging
by: Zhang, Haobo, et al.
Published: (2025)

On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
by: Terzić, Aleksandar, et al.
Published: (2024)

Transformer Circuit Faithfulness Metrics are not Robust
by: Miller, Joseph, et al.
Published: (2024)

QOG:Question and Options Generation based on Language Model
by: Zhou, Jincheng
Published: (2024)