Image de couverture de livre

BlackMamba: Mixture of Experts for State-Space Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Anthony, Quentin, Tokpanov, Yury, Glorioso, Paolo, Millidge, Beren
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence Distributed, Parallel, and Cluster Computing Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference
par: Yao, Jinghan, et autres
Publié: (2024)

Zyda-2: a 5 Trillion Token High-Quality Dataset
par: Tokpanov, Yury, et autres
Publié: (2024)

Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design
par: Anthony, Quentin, et autres
Publié: (2025)

Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Experts
par: Sun, Weigao, et autres
Publié: (2025)

FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
par: Jiang, Juyong, et autres
Publié: (2026)

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models
par: Xue, Fuzhao, et autres
Publié: (2024)

Zamba: A Compact 7B SSM Hybrid Model
par: Glorioso, Paolo, et autres
Publié: (2024)

A Survey on Inference Optimization Techniques for Mixture of Experts Models
par: Liu, Jiacheng, et autres
Publié: (2024)

Optimal Transport Aggregation for Distributed Mixture-of-Experts
par: Chamroukhi, Faïcel, et autres
Publié: (2023)

Utility-Driven Speculative Decoding for Mixture-of-Experts
par: Saxena, Anish, et autres
Publié: (2025)

Scalable Pretraining of Large Mixture of Experts Language Models on Aurora Super Computer
par: Vooturi, Dharma Teja, et autres
Publié: (2026)

Comet: Fine-grained Computation-communication Overlapping for Mixture-of-Experts
par: Zhang, Shulai, et autres
Publié: (2025)

RaMP: Runtime-Aware Megakernel Polymorphism for Mixture-of-Experts
par: Sharma, Vyom, et autres
Publié: (2026)

Scalable Training of Mixture-of-Experts Models with Megatron Core
par: Yan, Zijie, et autres
Publié: (2026)

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
par: Cai, Weilin, et autres
Publié: (2024)

Mixture of Experts with Mixture of Precisions for Tuning Quality of Service
par: Imani, HamidReza, et autres
Publié: (2024)

From Score Distributions to Balance: Plug-and-Play Mixture-of-Experts Routing
par: Shahout, Rana, et autres
Publié: (2025)

Mosaic: Data-Free Knowledge Distillation via Mixture-of-Experts for Heterogeneous Distributed Environments
par: Liu, Junming, et autres
Publié: (2025)

The Zamba2 Suite: Technical Report
par: Glorioso, Paolo, et autres
Publié: (2024)

Advancing MoE Efficiency: A Collaboration-Constrained Routing (C2R) Strategy for Better Expert Parallelism Design
par: Zhang, Mohan, et autres
Publié: (2025)

Make Every Draft Count: Hidden State based Speculative Decoding
par: Chen, Yuetao, et autres
Publié: (2026)

Accelerating MoE Model Inference with Expert Sharding
par: Balmau, Oana, et autres
Publié: (2025)

Fiddler: CPU-GPU Orchestration for Fast Inference of Mixture-of-Experts Models
par: Kamahori, Keisuke, et autres
Publié: (2024)

WDMoE: Wireless Distributed Mixture of Experts for Large Language Models
par: Xue, Nan, et autres
Publié: (2024)

Epistemic Observability in Language Models
par: Mason, Tony, et autres
Publié: (2026)

X-MoE: Enabling Scalable Training for Emerging Mixture-of-Experts Architectures on HPC Platforms
par: Yuan, Yueming, et autres
Publié: (2025)

Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scale
par: Ku, Jerome, et autres
Publié: (2025)

Efficient Deployment of Large Language Models on Resource-constrained Devices
par: Yao, Zhiwei, et autres
Publié: (2025)

PHOTON: Hierarchical Autoregressive Modeling for Lightspeed and Memory-Efficient Language Generation
par: Ichikawa, Yuma, et autres
Publié: (2025)

Minions: Cost-efficient Collaboration Between On-device and Cloud Language Models
par: Narayan, Avanika, et autres
Publié: (2025)

Characterizing and Understanding Energy Footprint and Efficiency of Small Language Model on Edges
par: Islam, Md Romyull, et autres
Publié: (2025)

ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation
par: Mei, Zhiyu, et autres
Publié: (2024)

Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models
par: Liu, Ji, et autres
Publié: (2024)

DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model
par: Gao, Chao, et autres
Publié: (2024)

A Sparsity Predicting Approach for Large Language Models via Activation Pattern Clustering
par: Dhar, Nobel, et autres
Publié: (2025)

Distributed Speculative Inference (DSI): Speculation Parallelism for Provably Faster Lossless Language Model Inference
par: Timor, Nadav, et autres
Publié: (2024)

Toward Sustainable GenAI using Generation Directives for Carbon-Friendly Large Language Model Inference
par: Li, Baolin, et autres
Publié: (2024)

FinGPT-HPC: Efficient Pretraining and Finetuning Large Language Models for Financial Applications with High-Performance Computing
par: Liu, Xiao-Yang, et autres
Publié: (2024)

SYMI: Efficient Mixture-of-Experts Training via Model and Optimizer State Decoupling
par: Skiadopoulos, Athinagoras, et autres
Publié: (2025)

HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission
par: Yang, Weihao, et autres
Publié: (2025)