Cover Image

Breaking Symmetry When Training Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zuo, Chunsheng, Guerzhoy, Michael
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Position Information Emerges in Causal Transformers Without Positional Encodings via Similarity of Nearby Embeddings
by: Zuo, Chunsheng, et al.
Published: (2024)

Rank, Head-Channel Non-Identifiability, and Symmetry Breaking: A Precise Analysis of Representational Collapse in Transformers
by: Cirrincione, Giansalvo
Published: (2026)

Process Supervision of Confidence Margin for Calibrated LLM Reasoning
by: Wang, Liaoyaqi, et al.
Published: (2026)

SPARKLING: Balancing Signal Preservation and Symmetry Breaking for Width-Progressive Learning
by: Yu, Qifan, et al.
Published: (2026)

Learning Adapter Rank via Symmetry Breaking
by: Doyle, Cooper, et al.
Published: (2025)

Breaking MLPerf Training: A Case Study on Optimizing BERT
by: Kim, Yongdeok, et al.
Published: (2024)

Single Character Perturbations Break LLM Alignment
by: Lin, Leon, et al.
Published: (2024)

Thinking into the Future: Latent Lookahead Training for Transformers
by: Noci, Lorenzo, et al.
Published: (2026)

When Can Transformers Count to n?
by: Yehudai, Gilad, et al.
Published: (2024)

When to Ponder: Adaptive Compute Allocation for Code Generation via Test-Time Training
by: Sim, Gihyeon
Published: (2025)

When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment
by: Xiao, Yuxin, et al.
Published: (2025)

On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capability
by: Zheng, Chenyu, et al.
Published: (2024)

Gated Linear Attention Transformers with Hardware-Efficient Training
by: Yang, Songlin, et al.
Published: (2023)

Breaking the Attention Bottleneck
by: Hilsenbek, Kalle
Published: (2024)

Breaking through the learning plateaus of in-context learning in Transformer
by: Fu, Jingwen, et al.
Published: (2023)

Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States
by: Yuan, Yurun, et al.
Published: (2026)

Breaking Training Bottlenecks: Effective and Stable Reinforcement Learning for Coding Models
by: Li, Zongqian, et al.
Published: (2026)

Occam's Razor and Bender and Koller's Octopus
by: Guerzhoy, Michael
Published: (2024)

Development of Pre-Trained Transformer-based Models for the Nepali Language
by: Thapa, Prajwal, et al.
Published: (2024)

A Multi-Level Framework for Accelerating Training Transformer Models
by: Zou, Longwei, et al.
Published: (2024)

Early Transformers: A study on Efficient Training of Transformer Models through Early-Bird Lottery Tickets
by: Cheekati, Shravan
Published: (2024)

When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars
by: Higuchi, Rei, et al.
Published: (2025)

Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior
by: Huang, Zeyi, et al.
Published: (2026)

Rationalizing Transformer Predictions via End-To-End Differentiable Self-Training
by: Brinner, Marc, et al.
Published: (2025)

MUDDFormer: Breaking Residual Bottlenecks in Transformers via Multiway Dynamic Dense Connections
by: Xiao, Da, et al.
Published: (2025)

Training Nonlinear Transformers for Chain-of-Thought Inference: A Theoretical Generalization Analysis
by: Li, Hongkang, et al.
Published: (2024)

Quagmires in SFT-RL Post-Training: When High SFT Scores Mislead and What to Use Instead
by: Kang, Feiyang, et al.
Published: (2025)

MOOSE-Star: Unlocking Tractable Training for Scientific Discovery by Breaking the Complexity Barrier
by: Yang, Zonglin, et al.
Published: (2026)

When Fairness Isn't Statistical: The Limits of Machine Learning in Evaluating Legal Reasoning
by: Barale, Claire, et al.
Published: (2025)

Training Dynamics of Transformers to Recognize Word Co-occurrence via Gradient Flow Analysis
by: Yang, Hongru, et al.
Published: (2024)

STAT: Shrinking Transformers After Training
by: Flynn, Megan, et al.
Published: (2024)

Circuit Breaking: Removing Model Behaviors with Targeted Ablation
by: Li, Maximilian, et al.
Published: (2023)

How does Multi-Task Training Affect Transformer In-Context Capabilities? Investigations with Function Classes
by: Bhasin, Harmon, et al.
Published: (2024)

When Greedy Wins: Emergent Exploitation Bias in Meta-Bandit LLM Training
by: Chen, Sanxing, et al.
Published: (2025)

How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Bias
by: Huang, Ruiquan, et al.
Published: (2025)

State Stream Transformer (SST) V2: Parallel Training of Nonlinear Recurrence for Latent Space Reasoning
by: Aviss, Thea
Published: (2026)

Does Alignment Tuning Really Break LLMs' Internal Confidence?
by: Oh, Hongseok, et al.
Published: (2024)

Break the Sequential Dependency of LLM Inference Using Lookahead Decoding
by: Fu, Yichao, et al.
Published: (2024)

STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMs
by: Dong, Peijie, et al.
Published: (2024)

Uniform-Correct Policy Optimization: Breaking RLVR's Indifference to Diversity
by: Lochab, Anamika, et al.
Published: (2026)