Cover Image

Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhu, Hanlin, Huang, Baihe, Zhang, Shaolun, Jordan, Michael, Jiao, Jiantao, Tian, Yuandong, Russell, Stuart
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Emergence of Superposition: Unveiling the Training Dynamics of Chain of Continuous Thought
by: Zhu, Hanlin, et al.
Published: (2025)

Reasoning by Superposition: A Theoretical Perspective on Chain of Continuous Thought
by: Zhu, Hanlin, et al.
Published: (2025)

Towards Optimal Statistical Watermarking
by: Huang, Baihe, et al.
Published: (2023)

Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformers
by: Huang, Yixiao, et al.
Published: (2025)

On Representation Complexity of Model-based and Model-free Reinforcement Learning
by: Zhu, Hanlin, et al.
Published: (2023)

Efficient Prompt Caching via Embedding Similarity
by: Zhu, Hanlin, et al.
Published: (2024)

Token Assorted: Mixing Latent and Text Tokens for Improved Language Model Reasoning
by: Su, DiJia, et al.
Published: (2025)

Towards Anytime-Valid Statistical Watermarking
by: Huang, Baihe, et al.
Published: (2026)

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF
by: Zhu, Banghua, et al.
Published: (2024)

An Analysis and Mitigation of the Reversal Curse
by: Lv, Ang, et al.
Published: (2023)

GSM-Agent: Understanding Agentic Reasoning Using Controllable Environments
by: Zhu, Hanlin, et al.
Published: (2025)

Mitigating Reversal Curse in Large Language Models via Semantic-aware Permutation Training
by: Guo, Qingyan, et al.
Published: (2024)

Toward a Theory of Tokenization in LLMs
by: Rajaraman, Nived, et al.
Published: (2024)

Transformers Provably Learn to Internalize Chain-of-Thought
by: Huang, Yixiao, et al.
Published: (2026)

The Factorization Curse: Which Tokens You Predict Underlie the Reversal Curse and More
by: Kitouni, Ouail, et al.
Published: (2024)

Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspective
by: Gan, Zeyu, et al.
Published: (2024)

Composing Global Solutions to Reasoning Tasks via Algebraic Objects in Neural Nets
by: Tian, Yuandong
Published: (2024)

JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attention
by: Tian, Yuandong, et al.
Published: (2023)

Towards Analyzing and Understanding the Limitations of VAPO: A Theoretical Perspective
by: Shao, Jintian, et al.
Published: (2025)

How Do LLMs Perform Two-Hop Reasoning in Context?
by: Guo, Tianyu, et al.
Published: (2025)

Safe Learning Under Irreversible Dynamics via Asking for Help
by: Plaut, Benjamin, et al.
Published: (2025)

Is the Reversal Curse a Binding Problem? Uncovering Limitations of Transformers from a Basic Generalization Failure
by: Wang, Boshi, et al.
Published: (2025)

The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A"
by: Berglund, Lukas, et al.
Published: (2023)

Feature Resemblance: Towards a Theoretical Understanding of Analogical Reasoning in Transformers
by: Xu, Ruichen, et al.
Published: (2026)

Sample Complexity and Representation Ability of Test-time Scaling Paradigms
by: Huang, Baihe, et al.
Published: (2025)

Reverse Training to Nurse the Reversal Curse
by: Golovneva, Olga, et al.
Published: (2024)

Param$Δ$ for Direct Weight Mixing: Post-Train Large Language Model at Zero Cost
by: Cao, Sheng, et al.
Published: (2025)

From Bits to Rounds: Parallel Decoding with Exploration for Diffusion Language Models
by: Fu, Hengyu, et al.
Published: (2025)

LoCoCo: Dropping In Convolutions for Long Context Compression
by: Cai, Ruisi, et al.
Published: (2024)

Avoiding Catastrophe in Online Learning by Asking for Help
by: Plaut, Benjamin, et al.
Published: (2024)

Exploring the Reversal Curse and Other Deductive Logical Reasoning in BERT and GPT-Based Large Language Models
by: Wu, Da, et al.
Published: (2023)

How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias
by: Huang, Ruiquan, et al.
Published: (2025)

Curse of High Dimensionality Issue in Transformer for Long-context Modeling
by: Zhang, Shuhai, et al.
Published: (2025)

Provable Scaling Laws of Feature Emergence from Learning Dynamics of Grokking
by: Tian, Yuandong
Published: (2025)

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts
by: Feng, Jiahai, et al.
Published: (2024)

Monitoring Latent World States in Language Models with Propositional Probes
by: Feng, Jiahai, et al.
Published: (2024)

TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decoding
by: Sun, Hanshi, et al.
Published: (2024)

Towards Efficient Resume Understanding: A Multi-Granularity Multi-Modal Pre-Training Approach
by: Jiang, Feihu, et al.
Published: (2024)

The Position Curse: LLMs Struggle to Locate the Last Few Items in a List
by: Zhang, Zhanqi, et al.
Published: (2026)

Principled Reinforcement Learning with Human Feedback from Pairwise or $K$-wise Comparisons
by: Zhu, Banghua, et al.
Published: (2023)