Cover Image

Tuning Language Models by Mixture-of-Depths Ensemble

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Luo, Haoyan, Specia, Lucia
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

From Understanding to Utilization: A Survey on Explainability for Large Language Models
by: Luo, Haoyan, et al.
Published: (2024)

MoESD: Mixture of Experts Stable Diffusion to Mitigate Gender Bias
by: Wang, Guorun, et al.
Published: (2024)

Mixture-of-Depths Attention
by: Zhu, Lianghui, et al.
Published: (2026)

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models
by: Feng, Wenfeng, et al.
Published: (2024)

Routing-Aligned Fine-Tuning for Multilingual Downstream Tasks in Mixture-of-Experts Models
by: Deng, Guanzhi, et al.
Published: (2026)

Determine-Then-Ensemble: Necessity of Top-k Union for Large Language Model Ensembling
by: Yao, Yuxuan, et al.
Published: (2024)

MoA: Heterogeneous Mixture of Adapters for Parameter-Efficient Fine-Tuning of Large Language Models
by: Cao, Jie, et al.
Published: (2025)

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts
by: Li, Dengchun, et al.
Published: (2024)

Impact of Fine-Tuning Methods on Memorization in Large Language Models
by: Hou, Jie, et al.
Published: (2025)

SpecFuse: Ensembling Large Language Models via Next-Segment Prediction
by: Lv, Bo, et al.
Published: (2024)

M-Ped: Multi-Prompt Ensemble Decoding for Large Language Models
by: Guo, Jiaxin, et al.
Published: (2024)

SciDFM: A Large Language Model with Mixture-of-Experts for Science
by: Sun, Liangtai, et al.
Published: (2024)

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
by: Jiang, Fan, et al.
Published: (2026)

LAET: A Layer-wise Adaptive Ensemble Tuning Framework for Pretrained Language Models
by: Ahad, Jawad Ibn, et al.
Published: (2025)

LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
by: Zheng, Yaowei, et al.
Published: (2024)

Multi-Programming Language Ensemble for Code Generation in Large Language Model
by: Xue, Tengfei, et al.
Published: (2024)

DR-LoRA: Dynamic Rank LoRA for Fine-Tuning Mixture-of-Experts Models
by: Deng, Guanzhi, et al.
Published: (2026)

Prompt-based Depth Pruning of Large Language Models
by: Wee, Juyun, et al.
Published: (2025)

LLMSurgeon: Diagnosing Data Mixture of Large Language Models
by: Luo, Yaxin, et al.
Published: (2026)

Large Language Models Are Still Misled by Simple Bias Ensembles
by: Sun, Zhouhao, et al.
Published: (2025)

Ensemble Debates with Local Large Language Models for AI Alignment
by: Sarabamoun, Ephraiem
Published: (2025)

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer
by: Liu, Boan, et al.
Published: (2023)

Group then Scale: Dynamic Mixture-of-Experts Multilingual Language Model
by: Li, Chong, et al.
Published: (2025)

DND: Boosting Large Language Models with Dynamic Nested Depth
by: Chen, Tieyuan, et al.
Published: (2025)

GRAPHMOE: Amplifying Cognitive Depth of Mixture-of-Experts Network via Introducing Self-Rethinking Mechanism
by: Lv, Bo, et al.
Published: (2025)

Overtrained Language Models Are Harder to Fine-Tune
by: Springer, Jacob Mitchell, et al.
Published: (2025)

Memorization in Fine-Tuned Large Language Models
by: Savine, Danil
Published: (2025)

ELPO: Ensemble Learning Based Prompt Optimization for Large Language Models
by: Zhang, Qing, et al.
Published: (2025)

Multimodal Generative Engine Optimization: Rank Manipulation for Vision-Language Model Rankers
by: Du, Yixuan, et al.
Published: (2026)

Model Tuning or Prompt Tuning? A Study of Large Language Models for Clinical Concept and Relation Extraction
by: Peng, Cheng, et al.
Published: (2023)

RegMix: Data Mixture as Regression for Language Model Pre-training
by: Liu, Qian, et al.
Published: (2024)

Mixture of Reasonings: Teach Large Language Models to Reason with Adaptive Strategies
by: Xiong, Tao, et al.
Published: (2025)

To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models
by: Trad, Fouad, et al.
Published: (2024)

Revisiting Catastrophic Forgetting in Large Language Model Tuning
by: Li, Hongyu, et al.
Published: (2024)

Safety-Aware Fine-Tuning of Large Language Models
by: Choi, Hyeong Kyu, et al.
Published: (2024)

Phased Instruction Fine-Tuning for Large Language Models
by: Pang, Wei, et al.
Published: (2024)

Investigating Instruction Tuning Large Language Models on Graphs
by: Zhu, Kerui, et al.
Published: (2024)

Chip-Tuning: Classify Before Language Models Say
by: Zhu, Fangwei, et al.
Published: (2024)

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
by: Liu, Yilun, et al.
Published: (2025)

Flexible and Effective Mixing of Large Language Models into a Mixture of Domain Experts
by: Lee, Rhui Dih, et al.
Published: (2024)