Image de couverture de livre

Theory, Analysis, and Best Practices for Sigmoid Self-Attention

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Ramapuram, Jason, Danieli, Federico, Dhekane, Eeshan, Weers, Floris, Busbridge, Dan, Ablin, Pierre, Likhomanenko, Tatiana, Digani, Jagrit, Gu, Zijin, Shidani, Amitis, Webb, Russ
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Scaling Properties of Continuous Diffusion Spoken Language Models
par: Ramapuram, Jason, et autres
Publié: (2026)

Poly-View Contrastive Learning
par: Shidani, Amitis, et autres
Publié: (2024)

Distillation Scaling Laws
par: Busbridge, Dan, et autres
Publié: (2025)

Revisiting the Scaling Properties of Downstream Metrics in Large Language Model Training
par: Krajewski, Jakub, et autres
Publié: (2025)

Path-Constrained Mixture-of-Experts
par: Gu, Zijin, et autres
Publié: (2026)

Completed Hyperparameter Transfer across Modules, Width, Depth, Batch and Duration
par: Mlodozeniec, Bruno, et autres
Publié: (2025)

A Small-Scale System for Autoregressive Program Synthesis Enabling Controlled Experimentation
par: Webb, Russ, et autres
Publié: (2026)

Attention to Mamba: A Recipe for Cross-Architecture Distillation
par: Moudgil, Abhinav, et autres
Publié: (2026)

Scaling Categorical Flow Maps
par: Davis, Oscar, et autres
Publié: (2026)

Ranking In Generalized Linear Bandits
par: Shidani, Amitis, et autres
Publié: (2022)

Which Evaluation for Which Model? A Taxonomy for Speech Model Assessment
par: de Seyssel, Maureen, et autres
Publié: (2025)

Omni-Router: Sharing Routing Decisions in Sparse Mixture-of-Experts for Speech Recognition
par: Gu, Zijin, et autres
Publié: (2025)

Locking Pretrained Weights via Deep Low-Rank Residual Distillation
par: Sakamoto, Keitaro, et autres
Publié: (2026)

Bernstein-type dimension-free concentration for self-normalised martingales
par: Akhavan, Arya, et autres
Publié: (2025)

The Design Space of Tri-Modal Masked Diffusion Models
par: Bethune, Louis, et autres
Publié: (2026)

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection
par: Bethune, Louis, et autres
Publié: (2025)

Beyond Real Data: Synthetic Data through the Lens of Regularization
par: Shidani, Amitis, et autres
Publié: (2025)

SpeakStream: Streaming Text-to-Speech with Interleaved Data
par: Bai, Richard He, et autres
Publié: (2025)

Scaling Laws for Optimal Data Mixtures
par: Shukor, Mustafa, et autres
Publié: (2025)

How Smooth Is Attention?
par: Castin, Valérie, et autres
Publié: (2023)

How PARTs assemble into wholes: Learning the relative composition of images
par: Ayoughi, Melika, et autres
Publié: (2025)

dMel: Speech Tokenization made Simple
par: Bai, Richard He, et autres
Publié: (2024)

Revisiting ASR Error Correction with Specialized Models
par: Gu, Zijin, et autres
Publié: (2024)

Learning Unmasking Policies for Diffusion Language Models
par: Jazbec, Metod, et autres
Publié: (2025)

Unsupervised ASR via Cross-Lingual Pseudo-Labeling
par: Likhomanenko, Tatiana, et autres
Publié: (2023)

Nectar: Neural Estimation of Cached-Token Attention via Regression
par: Monteiro, João, et autres
Publié: (2026)

SelfReflect: Can LLMs Communicate Their Internal Answer Distribution?
par: Kirchhof, Michael, et autres
Publié: (2025)

BugsRepo: A Comprehensive Curated Dataset of Bug Reports, Comments and Contributors Information from Bugzilla
par: Acharya, Jagrit, et autres
Publié: (2025)

Can We Enhance Bug Report Quality Using LLMs?: An Empirical Study of LLM-Based Bug Report Generation
par: Acharya, Jagrit, et autres
Publié: (2025)

Automatisierte Inhaltserschließung an der Bibliothek des Max-Planck-Instituts für Mathematik in den Naturwissenschaften
par: Weers, Beatrice Simon
Publié: (2026)

El dolor articular y su rela ción con las interferencias oclusales
par: Amitis Ruiseco Palomares
Publié: (2014)

OceanGliders Best Practices.
par: Testor, Pierre
Publié: (2018)

NeuroAssist: Enhancing Cognitive-Computer Synergy with Adaptive AI and Advanced Neural Decoding for Efficient EEG Signal Classification
par: Dandamudi, Eeshan G.
Publié: (2024)

Can External Validation Tools Improve Annotation Quality for LLM-as-a-Judge?
par: Findeis, Arduin, et autres
Publié: (2025)

Efficient Logistic Regression with Mixture of Sigmoids
par: Di Gennaro, Federico, et autres
Publié: (2026)

Closing the Gap Between Text and Speech Understanding in LLMs
par: Cuervo, Santiago, et autres
Publié: (2025)

El siglo del populismo. Historia, teoría, crítica
par: Silvana Ablin
Publié: (2023)

The AdEMAMix Optimizer: Better, Faster, Older
par: Pagliardini, Matteo, et autres
Publié: (2024)

Dynamic Gradient Alignment for Online Data Mixing
par: Fan, Simin, et autres
Publié: (2024)

Variable-Pitch-Propeller Mechanism Design, and Development of Heliquad for Mid-flight Flipping and Fault-Tolerant-Control
par: Kulkarni, Eeshan, et autres
Publié: (2024)