Image de couverture de livre

Scalable Efficient Training of Large Language Models with Low-dimensional Projected Attention

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Lv, Xingtai, Ding, Ning, Zhang, Kaiyan, Hua, Ermo, Cui, Ganqu, Zhou, Bowen
Format:	Preprint
Publié:	2024
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process
par: Hua, Ermo, et autres
Publié: (2024)

How to Synthesize Text Data without Model Collapse?
par: Zhu, Xuekai, et autres
Publié: (2024)

Technologies on Effectiveness and Efficiency: A Survey of State Spaces Models
par: Lv, Xingtai, et autres
Publié: (2025)

Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization
par: Hua, Ermo, et autres
Publié: (2024)

Towards a Unified View of Large Language Model Post-Training
par: Lv, Xingtai, et autres
Publié: (2025)

Fast and Slow Generating: An Empirical Study on Large and Small Language Models Collaborative Decoding
par: Zhang, Kaiyan, et autres
Publié: (2024)

Post-Trained MoE Can Skip Half Experts via Self-Distillation
par: Lv, Xingtai, et autres
Publié: (2026)

Mastering Text, Code and Math Simultaneously via Fusing Highly Specialized Language Models
par: Ding, Ning, et autres
Publié: (2024)

FlowRL: Matching Reward Distributions for LLM Reasoning
par: Zhu, Xuekai, et autres
Publié: (2025)

MedXpertQA: Benchmarking Expert-Level Medical Reasoning and Understanding
par: Zuo, Yuxin, et autres
Publié: (2025)

A Survey of Reinforcement Learning for Large Reasoning Models
par: Zhang, Kaiyan, et autres
Publié: (2025)

Sliding Window Attention Training for Efficient Large Language Models
par: Fu, Zichuan, et autres
Publié: (2025)

CoGenesis: A Framework Collaborating Large and Small Language Models for Secure Context-Aware Instruction Following
par: Zhang, Kaiyan, et autres
Publié: (2024)

Process Reinforcement through Implicit Rewards
par: Cui, Ganqu, et autres
Publié: (2025)

The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models
par: Cui, Ganqu, et autres
Publié: (2025)

SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning
par: Li, Haozhan, et autres
Publié: (2025)

UltraFeedback: Boosting Language Models with Scaled AI Feedback
par: Cui, Ganqu, et autres
Publié: (2023)

Train Small, Infer Large: Memory-Efficient LoRA Training for Large Language Models
par: Zhang, Jun, et autres
Publié: (2025)

Explicit Multi-head Attention for Inter-head Interaction in Large Language Models
par: Peng, Runyu, et autres
Publié: (2026)

EE-Tuning: An Economical yet Scalable Solution for Tuning Early-Exit Large Language Models
par: Pan, Xuchen, et autres
Publié: (2024)

YaRN: Efficient Context Window Extension of Large Language Models
par: Peng, Bowen, et autres
Publié: (2023)

EfficientQAT: Efficient Quantization-Aware Training for Large Language Models
par: Chen, Mengzhao, et autres
Publié: (2024)

A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithms
par: Gong, Ruihao, et autres
Publié: (2024)

LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Models
par: Yang, Yifan, et autres
Publié: (2024)

Automating Exploratory Proteomics Research via Language Models
par: Ding, Ning, et autres
Publié: (2024)

Automating Exploratory Multiomics Research via Language Models
par: Qu, Shang, et autres
Publié: (2025)

Training-Free Bayesianization for Low-Rank Adapters of Large Language Models
par: Shi, Haizhou, et autres
Publié: (2024)

APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models
par: Guan, Ziyi, et autres
Publié: (2024)

UltraMedical: Building Specialized Generalists in Biomedicine
par: Zhang, Kaiyan, et autres
Publié: (2024)

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing
par: Qi, Biqing, et autres
Publié: (2024)

Efficient Attention Mechanisms for Large Language Models: A Survey
par: Sun, Yutao, et autres
Publié: (2025)

QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models
par: Liu, Jing, et autres
Publié: (2023)

SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Models
par: Yang, Yu, et autres
Publié: (2024)

Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference
par: Samplawski, Colin, et autres
Publié: (2025)

Advancing LLM Reasoning Generalists with Preference Trees
par: Yuan, Lifan, et autres
Publié: (2024)

I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models
par: Hu, Xing, et autres
Publié: (2024)

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
par: Xiao, Guangxuan, et autres
Publié: (2022)

Learning to Reason as Action Abstractions with Scalable Mid-Training RL
par: Zhang, Shenao, et autres
Publié: (2025)

LocMoE: A Low-Overhead MoE for Large Language Model Training
par: Li, Jing, et autres
Publié: (2024)

EfficientLLM: Efficiency in Large Language Models
par: Yuan, Zhengqing, et autres
Publié: (2025)