Cover Image

Barriers to Universal Reasoning With Transformers (And How to Overcome Them)

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kraus, Oliver, Sarrof, Yash, Yao, Yuekun, Koller, Alexander, Hahn, Michael
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

On the Ability of Transformers to Verify Plans
by: Sarrof, Yash, et al.
Published: (2026)

The Expressive Capacity of State Space Models: A Formal Language Perspective
by: Sarrof, Yash, et al.
Published: (2024)

Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities
by: Jobanputra, Mayank, et al.
Published: (2025)

Loop, Think, & Generalize: Implicit Reasoning in Recurrent-Depth Transformers
by: Kohli, Harsh, et al.
Published: (2026)

Simple and effective data augmentation for compositional generalization
by: Yao, Yuekun, et al.
Published: (2024)

Predicting generalization performance with correctness discriminators
by: Yao, Yuekun, et al.
Published: (2023)

Reason to Rote: Rethinking Memorization in Reasoning
by: Du, Yupei, et al.
Published: (2025)

Language models can learn implicit multi-hop reasoning, but only if they have lots of training data
by: Yao, Yuekun, et al.
Published: (2025)

Reasoning Inconsistencies and How to Mitigate Them in Deep Learning
by: Arakelyan, Erik
Published: (2025)

Out-of-Context Reasoning in Large Language Models
by: Shaki, Jonathan, et al.
Published: (2025)

Emergent Stack Representations in Modeling Counter Languages Using Transformers
by: Tiwari, Utkarsh, et al.
Published: (2025)

A Formal Framework for Understanding Length Generalization in Transformers
by: Huang, Xinting, et al.
Published: (2024)

Enhancing Domain-Specific Encoder Models with LLM-Generated Data: How to Leverage Ontologies, and How to Do Without Them
by: Brinner, Marc, et al.
Published: (2025)

Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Them
by: Haresamudram, Harish, et al.
Published: (2024)

How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines
by: Sengupta, Ayan, et al.
Published: (2025)

Fantastic Reasoning Behaviors and Where to Find Them: Unsupervised Discovery of the Reasoning Process
by: Zhang, Zhenyu, et al.
Published: (2025)

One Sample to Rule Them All: Extreme Data Efficiency in Multidiscipline Reasoning with Reinforcement Learning
by: Li, Yiyuan, et al.
Published: (2026)

Understanding Contextual Recall in Transformers: How Finetuning Enables In-Context Reasoning over Pretraining Knowledge
by: Vasudeva, Bhavya, et al.
Published: (2026)

LinguaMap: Which Layers of LLMs Speak Your Language and How to Tune Them?
by: Tamo, J. Ben, et al.
Published: (2026)

UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs
by: Sinha, Yash, et al.
Published: (2024)

High-Stakes Personalization: Rethinking LLM Customization for Individual Investor Decision-Making
by: Sawant, Yash Ganpat
Published: (2026)

Reasoning in Transformers -- Mitigating Spurious Correlations and Reasoning Shortcuts
by: Enström, Daniel, et al.
Published: (2024)

Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers
by: Huang, Yixiao, et al.
Published: (2025)

Discovering Interpretable Algorithms by Decompiling Transformers to RASP
by: Huang, Xinting, et al.
Published: (2026)

Low-Perplexity LLM-Generated Sequences and Where To Find Them
by: Wuhrmann, Arthur, et al.
Published: (2025)

Understanding the Emergence of Seemingly Useless Features in Next-Token Predictors
by: Rofin, Mark, et al.
Published: (2026)

Explaining Text Similarity in Transformer Models
by: Vasileiou, Alexandros, et al.
Published: (2024)

Transformers meet Neural Algorithmic Reasoners
by: Bounsi, Wilfried, et al.
Published: (2024)

Compute Where it Counts: Self Optimizing Language Models
by: Akhauri, Yash, et al.
Published: (2026)

An Analysis of Embedding Layers and Similarity Scores using Siamese Neural Networks
by: Bingi, Yash, et al.
Published: (2023)

Transformers are Universal In-context Learners
by: Furuya, Takashi, et al.
Published: (2024)

Transformer-based Joint Modelling for Automatic Essay Scoring and Off-Topic Detection
by: Das, Sourya Dipta, et al.
Published: (2024)

WET: Overcoming Paraphrasing Vulnerabilities in Embeddings-as-a-Service with Linear Transformation Watermarks
by: Shetty, Anudeex, et al.
Published: (2024)

Fantastic Bugs and Where to Find Them in AI Benchmarks
by: Truong, Sang, et al.
Published: (2025)

Transformer See, Transformer Do: Copying as an Intermediate Step in Learning Analogical Reasoning
by: Hellwig, Philipp, et al.
Published: (2026)

Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learning
by: Huang, Xinting, et al.
Published: (2025)

SmartSwitch: Advancing LLM Reasoning by Overcoming Underthinking via Promoting Deeper Thought Exploration
by: Zhang, Xichen, et al.
Published: (2025)

Ensemble Self-Training for Unsupervised Machine Translation
by: Aharon, Ido, et al.
Published: (2026)

Contextualize-then-Aggregate: Circuits for In-Context Learning in Gemma-2 2B
by: Bakalova, Aleksandra, et al.
Published: (2025)

How Powerful are Decoder-Only Transformer Neural Models?
by: Roberts, Jesse
Published: (2023)