Copertina

OTCE: Hybrid SSM and Attention with Cross Domain Mixture of Experts to construct Observer-Thinker-Conceiver-Expresser

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Shi, Jingze, Xie, Ting, Wu, Bingheng, Zheng, Chunjun, Wang, Kai
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Artificial Intelligence
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Wonderful Matrices: Combining for a More Efficient and Effective Foundation Model Architecture
di: Shi, Jingze, et al.
Pubblicazione: (2024)

TransXSSM: A Hybrid Transformer State Space Model with Unified Rotary Position Embedding
di: Wu, Bingheng, et al.
Pubblicazione: (2025)

Trainable Dynamic Mask Sparse Attention
di: Shi, Jingze, et al.
Pubblicazione: (2025)

OmniMoE: An Efficient MoE by Orchestrating Atomic Experts at Scale
di: Shi, Jingze, et al.
Pubblicazione: (2026)

Pre-Attention Expert Prediction and Prefetching for Mixture-of-Experts Large Language Models
di: Zhu, Shien, et al.
Pubblicazione: (2025)

Wonderful Matrices: More Efficient and Effective Architecture for Language Modeling Tasks
di: Shi, Jingze, et al.
Pubblicazione: (2024)

Yuan 2.0-M32: Mixture of Experts with Attention Router
di: Wu, Shaohua, et al.
Pubblicazione: (2024)

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment
di: Liu, Zhili, et al.
Pubblicazione: (2024)

Mixture of Heterogeneous Grouped Experts for Language Modeling
di: Ma, Zhicheng, et al.
Pubblicazione: (2026)

Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
di: Lee, Rhui Dih, et al.
Pubblicazione: (2024)

Zamba: A Compact 7B SSM Hybrid Model
di: Glorioso, Paolo, et al.
Pubblicazione: (2024)

ExpertFlow: Efficient Mixture-of-Experts Inference via Predictive Expert Caching and Token Scheduling
di: He, Xin, et al.
Pubblicazione: (2024)

Multi-Head Mixture-of-Experts
di: Wu, Xun, et al.
Pubblicazione: (2024)

Mixture-of-Depths Attention
di: Zhu, Lianghui, et al.
Pubblicazione: (2026)

Knowledge Localization in Mixture-of-Experts LLMs Using Cross-Lingual Inconsistency
di: Bandarkar, Lucas, et al.
Pubblicazione: (2026)

MoSEs: Uncertainty-Aware AI-Generated Text Detection via Mixture of Stylistics Experts with Conditional Thresholds
di: Wu, Junxi, et al.
Pubblicazione: (2025)

SciDFM: A Large Language Model with Mixture-of-Experts for Science
di: Sun, Liangtai, et al.
Pubblicazione: (2024)

Routing by Analogy: kNN-Augmented Expert Assignment for Mixture-of-Experts
di: Lyu, Boxuan, et al.
Pubblicazione: (2026)

LD-MoLE: Learnable Dynamic Routing for Mixture of LoRA Experts
di: Zhuang, Yuan, et al.
Pubblicazione: (2025)

DynMoLE: Boosting Mixture of LoRA Experts Fine-Tuning with a Hybrid Routing Mechanism
di: Li, Dengchun, et al.
Pubblicazione: (2025)

SAMoRA: Semantic-Aware Mixture of LoRA Experts for Task-Adaptive Learning
di: Shi, Boyan, et al.
Pubblicazione: (2026)

MultiPL-MoE: Multi-Programming-Lingual Extension of Large Language Models through Hybrid Mixture-of-Experts
di: Wang, Qing, et al.
Pubblicazione: (2025)

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM
di: Sukhbaatar, Sainbayar, et al.
Pubblicazione: (2024)

Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
di: Deng, Guanzhi, et al.
Pubblicazione: (2026)

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
di: Jiang, Fan, et al.
Pubblicazione: (2026)

Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
di: Li, Houyi, et al.
Pubblicazione: (2025)

MoST: Mixing Speech and Text with Modality-Aware Mixture of Experts
di: Lou, Yuxuan, et al.
Pubblicazione: (2026)

Routing-Free Mixture-of-Experts
di: Liu, Yilun, et al.
Pubblicazione: (2026)

Multilingual Routing in Mixture-of-Experts
di: Bandarkar, Lucas, et al.
Pubblicazione: (2025)

SparseDoctor: Towards Efficient Chat Doctor with Mixture of Experts Enhanced Large Language Models
di: Zhang, Jianbin, et al.
Pubblicazione: (2025)

Concise Reasoning, Big Gains: Pruning Long Reasoning Trace with Difficulty-Aware Prompting
di: Wu, Yifan, et al.
Pubblicazione: (2025)

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
di: Gu, Naibin, et al.
Pubblicazione: (2025)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
di: Zhao, Guoliang, et al.
Pubblicazione: (2025)

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
di: Zhuang, Haomin, et al.
Pubblicazione: (2024)

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
di: Liu, Boan, et al.
Pubblicazione: (2023)

Superposition in Transformers: A Novel Way of Building Mixture of Experts
di: Chaliah, Ayoub Ben, et al.
Pubblicazione: (2024)

ConstitutionalExperts: Training a Mixture of Principle-based Prompts
di: Petridis, Savvas, et al.
Pubblicazione: (2024)

Mixture-of-Experts with Intermediate CTC Supervision for Accented Speech Recognition
di: Lee, Wonjun, et al.
Pubblicazione: (2026)

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
di: Li, Chong, et al.
Pubblicazione: (2025)

MMBERT: Scaled Mixture-of-Experts Multimodal BERT for Robust Chinese Hate Speech Detection under Cloaking Perturbations
di: Xue, Qiyao, et al.
Pubblicazione: (2025)