Cover Image

Mixture-of-Transformers: A Sparse and Scalable Architecture for Multi-Modal Foundation Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liang, Weixin, Yu, Lili, Luo, Liang, Iyer, Srinivasan, Dong, Ning, Zhou, Chunting, Ghosh, Gargi, Lewis, Mike, Yih, Wen-tau, Zettlemoyer, Luke, Lin, Xi Victoria
Format:	Preprint
Published:	2024
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts
by: Lin, Xi Victoria, et al.
Published: (2024)

Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity
by: Liang, Weixin, et al.
Published: (2025)

Memory Layers at Scale
by: Berges, Vincent-Pierre, et al.
Published: (2024)

LMFusion: Adapting Pretrained Language Models for Multimodal Generation
by: Shi, Weijia, et al.
Published: (2024)

Byte Latent Transformer: Patches Scale Better Than Tokens
by: Pagnoni, Artidoro, et al.
Published: (2024)

Improving Factuality with Explicit Working Memory
by: Chen, Mingda, et al.
Published: (2024)

Instruction-tuned Language Models are Better Knowledge Learners
by: Jiang, Zhengbao, et al.
Published: (2024)

Continual Learning via Sparse Memory Finetuning
by: Lin, Jessy, et al.
Published: (2025)

Fast Byte Latent Transformer
by: Kallini, Julie, et al.
Published: (2026)

Latent Speech-Text Transformer
by: Lu, Yen-Ju, et al.
Published: (2025)

Compute Optimal Tokenization
by: Limisiewicz, Tomasz, et al.
Published: (2026)

Anchored Decoding: Provably Reducing Copyright Risk for Any Language Model
by: He, Jacqueline, et al.
Published: (2026)

In-context Pretraining: Language Modeling Beyond Document Boundaries
by: Shi, Weijia, et al.
Published: (2023)

Reliable, Adaptable, and Attributable Language Models with Retrieval
by: Asai, Akari, et al.
Published: (2024)

ImpRAG: Retrieval-Augmented Generation with Implicit Queries
by: Zhang, Wenzheng, et al.
Published: (2025)

Self-Alignment with Instruction Backtranslation
by: Li, Xian, et al.
Published: (2023)

HoneyBee: Data Recipes for Vision-Language Reasoners
by: Bansal, Hritik, et al.
Published: (2025)

Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering
by: Chen, Mingda, et al.
Published: (2023)

CAT: Content-Adaptive Image Tokenization
by: Shen, Junhong, et al.
Published: (2025)

Learning to Reason for Factuality
by: Chen, Xilun, et al.
Published: (2025)

Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model
by: Zhou, Chunting, et al.
Published: (2024)

Altogether: Image Captioning via Re-aligning Alt-text
by: Xu, Hu, et al.
Published: (2024)

DRAMA: Diverse Augmentation from Large Language Models to Smaller Dense Retrievers
by: Ma, Xueguang, et al.
Published: (2025)

ReasonIR: Training Retrievers for Reasoning Tasks
by: Shao, Rulin, et al.
Published: (2025)

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
by: Sukhbaatar, Sainbayar, et al.
Published: (2024)

Nearest Neighbor Speculative Decoding for LLM Generation and Attribution
by: Li, Minghan, et al.
Published: (2024)

Procedural Knowledge at Scale Improves Reasoning
by: Wu, Di, et al.
Published: (2026)

RA-DIT: Retrieval-Augmented Dual Instruction Tuning
by: Lin, Xi Victoria, et al.
Published: (2023)

Slicing and Dicing: Configuring Optimal Mixtures of Experts
by: Li, Margaret, et al.
Published: (2026)

ALMA: Alignment with Minimal Annotation
by: Yasunaga, Michihiro, et al.
Published: (2024)

The weighted Bergman spaces and complex reflection groups
by: Ghosh, Gargi
Published: (2021)

Computational Approaches to Understanding Large Language Model Impact on Writing and Information Ecosystems
by: Liang, Weixin
Published: (2025)

Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length
by: Ma, Xuezhe, et al.
Published: (2024)

Micro Language Models Enable Instant Responses
by: Cheng, Wen, et al.
Published: (2026)

Advances in Modal Regression: From Theoretical Foundations to Practical Implementations
by: Sijia Xiang, et al.
Published: (2026)

Learning Facts at Scale with Active Reading
by: Lin, Jessy, et al.
Published: (2025)

Group-Level Data Selection for Efficient Pretraining
by: Yu, Zichun, et al.
Published: (2025)

FACTORY: A Challenging Human-Verified Prompt Set for Long-Form Factuality
by: Chen, Mingda, et al.
Published: (2025)

Demystifying Prompts in Language Models via Perplexity Estimation
by: Gonen, Hila, et al.
Published: (2022)

l0-Regularized Sparse Coding-based Interpretable Network for Multi-Modal Image Fusion
by: Panda, Gargi, et al.
Published: (2024)