Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Masarczyk, Wojciech, Ostaszewski, Mateusz, Cheng, Tin Sum, Trzciński, Tomasz, Lucchi, Aurelien, Pascanu, Razvan
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Optimizer choice matters for the emergence of Neural Collapse
di: Zhao, Jim, et al.
Pubblicazione: (2026)

Characterizing Overfitting in Kernel Ridgeless Regression Through the Eigenspectrum
di: Cheng, Tin Sum, et al.
Pubblicazione: (2024)

A Comprehensive Analysis on the Learning Curve in Kernel Ridge Regression
di: Cheng, Tin Sum, et al.
Pubblicazione: (2024)

Overestimation, Overfitting, and Plasticity in Actor-Critic: the Bitter Lesson of Reinforcement Learning
di: Nauman, Michal, et al.
Pubblicazione: (2024)

No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPO
di: Moalla, Skander, et al.
Pubblicazione: (2024)

Softmax is not Enough (for Sharp Size Generalisation)
di: Veličković, Petar, et al.
Pubblicazione: (2024)

Sharp Generalization Bounds for Foundation Models with Asymmetric Randomized Low-Rank Adapters
di: Kratsios, Anastasis, et al.
Pubblicazione: (2025)

Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem
di: Wołczyk, Maciej, et al.
Pubblicazione: (2024)

Lattice: Learning to Efficiently Compress the Memory
di: Karami, Mahdi, et al.
Pubblicazione: (2025)

Latent Space Representations of Neural Algorithmic Reasoners
di: Mirjanić, Vladimir V., et al.
Pubblicazione: (2023)

Deep Grokking: Would Deep Neural Networks Generalize Better?
di: Fan, Simin, et al.
Pubblicazione: (2024)

A Case for Validation Buffer in Pessimistic Actor-Critic
di: Nauman, Michal, et al.
Pubblicazione: (2024)

NoProp: Training Neural Networks without Full Back-propagation or Full Forward-propagation
di: Li, Qinyu, et al.
Pubblicazione: (2025)

Exploring the Stability Gap in Continual Learning: The Role of the Classification Head
di: Łapacz, Wojciech, et al.
Pubblicazione: (2024)

A General Theory for Softmax Gating Multinomial Logistic Mixture of Experts
di: Nguyen, Huy, et al.
Pubblicazione: (2023)

Unbiased and Sign Compression in Distributed Learning: Comparing Noise Resilience via SDEs
di: Compagnoni, Enea Monzio, et al.
Pubblicazione: (2025)

Spectral Analysis of Molecular Kernels: When Richer Features Do Not Guarantee Better Generalization
di: Jamali, Asma, et al.
Pubblicazione: (2025)

Revisiting Supervision for Continual Representation Learning
di: Marczak, Daniel, et al.
Pubblicazione: (2023)

What Can Grokking Teach Us About Learning Under Nonstationarity?
di: Lyle, Clare, et al.
Pubblicazione: (2025)

Meta-learning how to Share Credit among Macro-Actions
di: Hosu, Ionel-Alexandru, et al.
Pubblicazione: (2025)

Revisiting Adam for Streaming Reinforcement Learning
di: Gogianu, Florin, et al.
Pubblicazione: (2026)

ELROND: Exploring and decomposing intrinsic capabilities of diffusion models
di: Skierś, Paweł, et al.
Pubblicazione: (2026)

Initial Guessing Bias: How Untrained Networks Favor Some Classes
di: Francazi, Emanuele, et al.
Pubblicazione: (2023)

How do language models learn facts? Dynamics, curricula and hallucinations
di: Zucchet, Nicolas, et al.
Pubblicazione: (2025)

Is Temporal Difference Learning the Gold Standard for Stitching in RL?
di: Bortkiewicz, Michał, et al.
Pubblicazione: (2025)

MISS: Multiclass Interpretable Scoring Systems
di: Grzeszczyk, Michal K., et al.
Pubblicazione: (2024)

A Theoretical Analysis of the Learning Dynamics under Class Imbalance
di: Francazi, Emanuele, et al.
Pubblicazione: (2022)

Theoretical characterisation of the Gauss-Newton conditioning in Neural Networks
di: Zhao, Jim, et al.
Pubblicazione: (2024)

Why Do We Need Warm-up? A Theoretical Perspective
di: Alimisis, Foivos, et al.
Pubblicazione: (2025)

MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
di: Karami, Mahdi, et al.
Pubblicazione: (2025)

Attention as a Hypernetwork
di: Schug, Simon, et al.
Pubblicazione: (2024)

Balancing Expressivity and Robustness: Constrained Rational Activations for Reinforcement Learning
di: Surdej, Rafał, et al.
Pubblicazione: (2025)

Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control
di: Nauman, Michal, et al.
Pubblicazione: (2024)

Benchmarking Pretrained Molecular Embedding Models For Molecular Representation Learning
di: Praski, Mateusz, et al.
Pubblicazione: (2025)

HyperSound: Generating Implicit Neural Representations of Audio Signals with Hypernetworks
di: Szatkowski, Filip, et al.
Pubblicazione: (2022)

On the Interaction of Batch Noise, Adaptivity, and Compression, under $(L_0,L_1)$-Smoothness: An SDE Approach
di: Compagnoni, Enea Monzio, et al.
Pubblicazione: (2025)

Hadamard product in deep learning: Introduction, Advances and Challenges
di: Chrysos, Grigorios G, et al.
Pubblicazione: (2025)

How to Train Your Multi-Exit Model? Analyzing the Impact of Training Strategies
di: Kubaty, Piotr, et al.
Pubblicazione: (2024)

Cubic regularized subspace Newton for non-convex optimization
di: Zhao, Jim, et al.
Pubblicazione: (2024)

Round and Round We Go! What makes Rotary Positional Encodings useful?
di: Barbero, Federico, et al.
Pubblicazione: (2024)