Cover Image

Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Tiberi, Lorenzo, Mignacco, Francesca, Irie, Kazuki, Sompolinsky, Haim
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Disordered Systems and Neural Networks Statistical Mechanics
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Optimal Protocols for Continual Learning via Statistical Physics and Control Theory
by: Mori, Francesco, et al.
Published: (2024)

Coding schemes in neural networks learning classification tasks
by: van Meegen, Alexander, et al.
Published: (2024)

Analytic theory of dropout regularization
by: Mori, Francesco, et al.
Published: (2025)

A statistical physics framework for optimal learning
by: Mignacco, Francesca, et al.
Published: (2025)

Nonlinear classification of neural manifolds with contextual information
by: Mignacco, Francesca, et al.
Published: (2024)

Interpreting the Synchronization Gap: The Hidden Mechanism Inside Diffusion Transformers
by: Albrychiewicz, Emil, et al.
Published: (2026)

Statistical Mechanics of the Sub-Optimal Transport
by: Piombo, Riccardo, et al.
Published: (2026)

Dissecting a Small Artificial Neural Network
by: Yang, Xiguang, et al.
Published: (2025)

The Physics of Data and Tasks: Theories of Locality and Compositionality in Deep Learning
by: Favero, Alessandro
Published: (2025)

Fast training and sampling of Restricted Boltzmann Machines
by: Béreux, Nicolas, et al.
Published: (2024)

Statistical Mechanics and Artificial Neural Networks: Principles, Models, and Applications
by: Böttcher, Lucas, et al.
Published: (2024)

Deterministic versus stochastic dynamical classifiers: opposing random adversarial attacks with noise
by: Chicchi, Lorenzo, et al.
Published: (2024)

Feature learning from non-Gaussian inputs: the case of Independent Component Analysis in high dimensions
by: Ricci, Fabiola, et al.
Published: (2025)

Testing the Localization Landscape Theory on the Bethe Lattice
by: Tonetti, Lorenzo, et al.
Published: (2025)

Uncovering statistical structure in large-scale neural activity with Restricted Boltzmann Machines
by: Béreux, Nicolas, et al.
Published: (2026)

Critical and Nonpercolating Phases in Bond Percolation on the Song-Havlin-Makse Network
by: Wataya, Kazuki, et al.
Published: (2025)

The Symmetric Perceptron: a Teacher-Student Scenario
by: Catania, Giovanni, et al.
Published: (2026)

Discrete generative diffusion models without stochastic differential equations: a tensor network approach
by: Causer, Luke, et al.
Published: (2024)

Weight fluctuations in (deep) linear neural networks and a derivation of the inverse-variance flatness relation
by: Gross, Markus, et al.
Published: (2023)

Two failure modes of deep transformers and how to avoid them: a unified theory of signal propagation at initialisation
by: Giorlandino, Alessio, et al.
Published: (2025)

Estimating Global Input Relevance and Enforcing Sparse Representations with a Scalable Spectral Neural Network Approach
by: Chicchi, Lorenzo, et al.
Published: (2024)

Stable Attractors for Neural networks classification via Ordinary Differential Equations (SA-nODE)
by: Marino, Raffaele, et al.
Published: (2023)

Berezinskii-Kosterlitz-Thouless-type Transition in Site Percolation on the Diamond Hierarchical Lattice
by: Hasegawa, Takehisa, et al.
Published: (2026)

The Copycat Perceptron: Smashing Barriers Through Collective Learning
by: Catania, Giovanni, et al.
Published: (2023)

Fundamental operating regimes, hyper-parameter fine-tuning and glassiness: towards an interpretable replica-theory for trained restricted Boltzmann machines
by: Fachechi, Alberto, et al.
Published: (2024)

The autoregressive neural network architecture of the Boltzmann distribution of pairwise interacting spins systems
by: Biazzo, Indaco
Published: (2023)

Distinct mechanisms underlying in-context learning in transformers
by: Gibson, Cole, et al.
Published: (2026)

Universal Scaling Laws of Absorbing Phase Transitions in Artificial Deep Neural Networks
by: Tamai, Keiichi, et al.
Published: (2023)

Inferring Higher-Order Couplings with Neural Networks
by: Decelle, Aurélien, et al.
Published: (2025)

Explaining the effects of non-convergent sampling in the training of Energy-Based Models
by: Agoritsas, Elisabeth, et al.
Published: (2023)

Factual recall in linear associative memories: sharp asymptotics and mechanistic insights
by: Giorlandino, Alessio, et al.
Published: (2026)

Sampling with flows, diffusion and autoregressive neural networks: A spin-glass perspective
by: Ghio, Davide, et al.
Published: (2023)

Stochastic Gradient Descent-like relaxation is equivalent to Metropolis dynamics in discrete optimization and inference problems
by: Angelini, Maria Chiara, et al.
Published: (2023)

Escape dynamics and implicit bias of one-pass SGD in overparameterized quadratic networks
by: Bocchi, Dario, et al.
Published: (2026)

Inferring effective couplings with Restricted Boltzmann Machines
by: Decelle, Aurélien, et al.
Published: (2023)

From Classical to Quantum: Extending Prometheus for Unsupervised Discovery of Phase Transitions in Three Dimensions and Quantum Systems
by: Yee, Brandon, et al.
Published: (2026)

Transfer Learning in $\ell_1$ Regularized Regression: Hyperparameter Selection Strategy based on Sharp Asymptotic Analysis
by: Okajima, Koki, et al.
Published: (2024)

A theoretical framework for overfitting in energy-based modeling
by: Catania, Giovanni, et al.
Published: (2025)

Uncertainty in AI-driven Monte Carlo simulations
by: Tzivrailis, Dimitrios, et al.
Published: (2025)

Dynamic neuron approach to deep neural networks: Decoupling neurons for renormalization group analysis
by: Lee, Donghee, et al.
Published: (2024)