Cover Image

How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Huang, Ruiquan, Liang, Yingbin, Yang, Jing
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Non-asymptotic Convergence of Training Transformers for Next-token Prediction
by: Huang, Ruiquan, et al.
Published: (2024)

Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis
by: Yang, Hongru, et al.
Published: (2024)

Robust Offline Reinforcement Learning for Non-Markovian Decision Processes
by: Huang, Ruiquan, et al.
Published: (2024)

Provably Efficient UCB-type Algorithms For Learning Predictive State Representations
by: Huang, Ruiquan, et al.
Published: (2023)

In-Context Learning with Representations: Contextual Generalization of Trained Transformers
by: Yang, Tong, et al.
Published: (2024)

Breaking the Computational Barrier: Provably Efficient Actor-Critic for Low-Rank MDPs
by: Huang, Ruiquan, et al.
Published: (2026)

Constraint-Rectified Training for Efficient Chain-of-Thought
by: Wu, Qinhang, et al.
Published: (2026)

Tracking the Feature Dynamics in LLM Training: A Mechanistic Study
by: Xu, Yang, et al.
Published: (2024)

Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamics
by: Zhu, Hanlin, et al.
Published: (2024)

Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis
by: Li, Hongkang, et al.
Published: (2024)

Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
by: Wang, Xiaolei, et al.
Published: (2024)

How Quantization Shapes Bias in Large Language Models
by: Marcuzzi, Federico, et al.
Published: (2025)

Implicit Optimization Bias of Next-Token Prediction in Linear Models
by: Thrampoulidis, Christos
Published: (2024)

Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Training
by: Rahmati, Elnaz, et al.
Published: (2026)

The Missing Half: Unveiling Training-time Implicit Safety Risks Beyond Deployment
by: Zhang, Zhexin, et al.
Published: (2026)

A Theoretical Analysis of Self-Supervised Learning for Vision Transformers
by: Huang, Yu, et al.
Published: (2024)

Comparative Study of Pre-Trained BERT and Large Language Models for Code-Mixed Named Entity Recognition
by: Shirke, Mayur, et al.
Published: (2025)

Theoretical Understanding of In-Context Learning in Shallow Transformers with Unstructured Data
by: Xing, Yue, et al.
Published: (2024)

Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models
by: Wang, Huazheng, et al.
Published: (2025)

How to Train Long-Context Language Models (Effectively)
by: Gao, Tianyu, et al.
Published: (2024)

From Lists to Emojis: How Format Bias Affects Model Alignment
by: Zhang, Xuanchang, et al.
Published: (2024)

Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
by: Li, Hongbo, et al.
Published: (2025)

How to Train Private Clinical Language Models: A Comparative Study of Privacy-Preserving Pipelines for ICD-9 Coding
by: Dufour, Mathieu, et al.
Published: (2025)

Reasoning Bias of Next Token Prediction Training
by: Lin, Pengxiao, et al.
Published: (2025)

CREAM: Consistency Regularized Self-Rewarding Language Models
by: Wang, Zhaoyang, et al.
Published: (2024)

Eliminating Position Bias of Language Models: A Mechanistic Approach
by: Wang, Ziqi, et al.
Published: (2024)

Development of Pre-Trained Transformer-based Models for the Nepali Language
by: Thapa, Prajwal, et al.
Published: (2024)

Causally Testing Gender Bias in LLMs: A Case Study on Occupational Bias
by: Chen, Yuen, et al.
Published: (2022)

DataFlex: A Unified Framework for Data-Centric Dynamic Training of Large Language Models
by: Liang, Hao, et al.
Published: (2026)

Demystifying Verbatim Memorization in Large Language Models
by: Huang, Jing, et al.
Published: (2024)

Safe Reinforcement Learning with Free-form Natural Language Constraints and Pre-Trained Language Models
by: Lou, Xingzhou, et al.
Published: (2024)

Advanced Multimodal Deep Learning Architecture for Image-Text Matching
by: Wang, Jinyin, et al.
Published: (2024)

When Does Removing LayerNorm Help? Activation Bounding as a Regime-Dependent Implicit Regularizer
by: Verma, Lucky
Published: (2026)

Bootstrapping Language Models with DPO Implicit Rewards
by: Chen, Changyu, et al.
Published: (2024)

How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes
by: Bhasin, Harmon, et al.
Published: (2024)

Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers
by: Chen, Siyu, et al.
Published: (2024)

Linear Dynamics in the RLVR Training of Large Language Models
by: Wang, Tianle, et al.
Published: (2026)

Training-Free Dynamic Upcycling of Expert Language Models
by: Fanì, Eros, et al.
Published: (2026)

Breaking Language Barriers or Reinforcing Bias? A Study of Gender and Racial Disparities in Multilingual Contrastive Vision Language Models
by: Sahili, Zahraa Al, et al.
Published: (2025)

Correcting Mean Bias in Text Embeddings: A Refined Renormalization with Training-Free Improvements on MMTEB
by: Ren, Xingyu, et al.
Published: (2025)