Copertina

Improving Reasoning Capabilities in Small Models through Mixture-of-Layers Distillation with Stepwise Attention on Key Information

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Chen, Yao, Sheng, Jiawei, Zhang, Wenyuan, Liu, Tingwen
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Exploring the System 1 Thinking Capability of Large Reasoning Models
di: Zhang, Wenyuan, et al.
Pubblicazione: (2025)

ATTNPO: Attention-Guided Process Supervision for Efficient Reasoning
di: Nie, Shuaiyi, et al.
Pubblicazione: (2026)

Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking
di: Zhang, Zefeng, et al.
Pubblicazione: (2024)

Distilling Mathematical Reasoning Capabilities into Small Language Models
di: Zhu, Xunyu, et al.
Pubblicazione: (2024)

Revealing and Mitigating the Challenge of Detecting Character Knowledge Errors in LLM Role-Playing
di: Zhang, Wenyuan, et al.
Pubblicazione: (2024)

Mixture of Hidden-Dimensions Transformer
di: Chen, Yilong, et al.
Pubblicazione: (2024)

Improving Mathematical Reasoning Capabilities of Small Language Models via Feedback-Driven Distillation
di: Zhu, Xunyu, et al.
Pubblicazione: (2024)

MoR: Mixture of Ranks for Low-Rank Adaptation Tuning
di: Tang, Chuanyu, et al.
Pubblicazione: (2024)

IBMEA: Exploring Variational Information Bottleneck for Multi-modal Entity Alignment
di: Su, Taoyu, et al.
Pubblicazione: (2024)

Unlocking the Power of Large Language Models for Multi-table Entity Matching
di: Tang, Yingkai, et al.
Pubblicazione: (2026)

HyperMem: Hypergraph Memory for Long-Term Conversations
di: Yue, Juwei, et al.
Pubblicazione: (2026)

LoginMEA: Local-to-Global Interaction Network for Multi-modal Entity Alignment
di: Su, Taoyu, et al.
Pubblicazione: (2024)

Neural-Symbolic Collaborative Distillation: Advancing Small Language Models for Complex Reasoning Tasks
di: Liao, Huanxuan, et al.
Pubblicazione: (2024)

SOTOPIA-$Ω$: Dynamic Strategy Injection Learning and Social Instruction Following Evaluation for Social Agents
di: Zhang, Wenyuan, et al.
Pubblicazione: (2025)

Distilling to Hybrid Attention Models via KL-Guided Layer Selection
di: Li, Yanhong, et al.
Pubblicazione: (2025)

Self-Enhanced Reasoning Training: Activating Latent Reasoning in Small Models for Enhanced Reasoning Distillation
di: Zhang, Yong, et al.
Pubblicazione: (2025)

Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck
di: Cui, Shiyao, et al.
Pubblicazione: (2023)

$\textit{SKIntern}$: Internalizing Symbolic Knowledge for Distilling Better CoT Capabilities into Small Language Models
di: Liao, Huanxuan, et al.
Pubblicazione: (2024)

Unveiling the Key Factors for Distilling Chain-of-Thought Reasoning
di: Chen, Xinghao, et al.
Pubblicazione: (2025)

Stepwise Informativeness Search for Efficient and Effective LLM Reasoning
di: Wang, Siyuan, et al.
Pubblicazione: (2025)

FFT: Towards Harmlessness Evaluation and Analysis for LLMs with Factuality, Fairness, Toxicity
di: Cui, Shiyao, et al.
Pubblicazione: (2023)

Routing Distilled Knowledge via Mixture of LoRA Experts for Large Language Model based Bundle Generation
di: Feng, Kaidong, et al.
Pubblicazione: (2025)

Adaptive Data Augmentation for Aspect Sentiment Quad Prediction
di: Zhang, Wenyuan, et al.
Pubblicazione: (2024)

MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Models
di: Chen, Justin Chih-Yao, et al.
Pubblicazione: (2024)

Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
di: Chen, Yilong, et al.
Pubblicazione: (2026)

Unveiling the Magic: Investigating Attention Distillation in Retrieval-augmented Generation
di: Li, Zizhong, et al.
Pubblicazione: (2024)

Enhancing Mathematical Reasoning in LLMs by Stepwise Correction
di: Wu, Zhenyu, et al.
Pubblicazione: (2024)

Enhancing Multilingual Capabilities of Large Language Models through Self-Distillation from Resource-Rich Languages
di: Zhang, Yuanchi, et al.
Pubblicazione: (2024)

Effectiveness of Chain-of-Thought in Distilling Reasoning Capability from Large Language Models
di: Do, Cong-Thanh, et al.
Pubblicazione: (2025)

Stepwise Self-Consistent Mathematical Reasoning with Large Language Models
di: Zhao, Zilong, et al.
Pubblicazione: (2024)

MIND: From Passive Mimicry to Active Reasoning through Capability-Aware Multi-Perspective CoT Distillation
di: Cui, Jin, et al.
Pubblicazione: (2026)

Satisfiability Solving with LLMs: A Matched-Pair Evaluation of Reasoning Capability
di: Zhang, Leizhen, et al.
Pubblicazione: (2026)

Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning
di: Lee, Sangmook, et al.
Pubblicazione: (2025)

Rethinking Stepwise Model Routing: A Cost-Efficient Table Reasoning Perspective
di: Ye, Shenghao, et al.
Pubblicazione: (2026)

A Stepwise-Enhanced Reasoning Framework for Large Language Models Based on External Subgraph Generation
di: Zhang, Xin, et al.
Pubblicazione: (2025)

SAT: Balancing Reasoning Accuracy and Efficiency with Stepwise Adaptive Thinking
di: Huang, Weiyang, et al.
Pubblicazione: (2026)

Mixture-of-Agents Enhances Large Language Model Capabilities
di: Wang, Junlin, et al.
Pubblicazione: (2024)

Med-PRM: Medical Reasoning Models with Stepwise, Guideline-verified Process Rewards
di: Yun, Jaehoon, et al.
Pubblicazione: (2025)

Debiasing Multimodal Large Language Models via Noise-Aware Preference Optimization
di: Zhang, Zefeng, et al.
Pubblicazione: (2025)

Dancing with Critiques: Enhancing LLM Reasoning with Stepwise Natural Language Self-Critique
di: Li, Yansi, et al.
Pubblicazione: (2025)