Cover Image

Pretraining A Large Language Model using Distributed GPUs: A Memory-Efficient Decentralized Paradigm

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Zhang, Jinrui, Xiao, Chaodong, Wu, Aoqi, Zhang, Xindong, Zhang, Lei
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

MLP Memory: A Retriever-Pretrained Memory for Large Language Models
by: Wei, Rubin, et al.
Published: (2025)

Give Us the Facts: Enhancing Large Language Models with Knowledge Graphs for Fact-aware Language Modeling
by: Yang, Linyao, et al.
Published: (2023)

Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
by: Cao, Jiaqi, et al.
Published: (2025)

DEP: A Decentralized Large Language Model Evaluation Protocol
by: Peng, Jianxiang, et al.
Published: (2026)

MINI-LLM: Memory-Efficient Structured Pruning for Large Language Models
by: Cheng, Hongrong, et al.
Published: (2024)

Unifying Large Language Models and Knowledge Graphs: A Roadmap
by: Pan, Shirui, et al.
Published: (2023)

UniVS: Unified and Universal Video Segmentation with Prompts as Queries
by: Li, Minghan, et al.
Published: (2024)

CodePMP: Scalable Preference Model Pretraining for Large Language Model Reasoning
by: Yu, Huimu, et al.
Published: (2024)

Efficient Pretraining Data Selection for Language Models via Multi-Actor Collaboration
by: Bai, Tianyi, et al.
Published: (2024)

Knowledge Dependency Estimation for Reliable Question Answering
by: Tong, Chaodong, et al.
Published: (2026)

HaluNet: Learning Hallucination Risk from Internal Signals in LLM Question Answering
by: Tong, Chaodong, et al.
Published: (2025)

The Role of Mixed-Language Documents for Multilingual Large Language Model Pretraining
by: Shao, Jiandong, et al.
Published: (2026)

Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspective
by: Yu, Yiyao, et al.
Published: (2025)

Reference Trustable Decoding: A Training-Free Augmentation Paradigm for Large Language Models
by: Shi, Luohe, et al.
Published: (2024)

Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Method
by: Zhang, Weichao, et al.
Published: (2024)

On The Role of Pretrained Language Models in General-Purpose Text Embeddings: A Survey
by: Zhang, Meishan, et al.
Published: (2025)

LLMVA-GEBC: Large Language Model with Video Adapter for Generic Event Boundary Captioning
by: Tang, Yolo Yunlong, et al.
Published: (2023)

A Tutorial on the Pretrain-Finetune Paradigm for Natural Language Processing
by: Wang, Yu, et al.
Published: (2024)

FinGPT-HPC: Efficient Pretraining and Finetuning Large Language Models for Financial Applications with High-Performance Computing
by: Liu, Xiao-Yang, et al.
Published: (2024)

Enhancing Knowledge Distillation of Large Language Models through Efficient Multi-Modal Distribution Alignment
by: Peng, Tianyu, et al.
Published: (2024)

Knowledge Reasoning Language Model: Unifying Knowledge and Language for Inductive Knowledge Graph Reasoning
by: Zhuo, Xingrui, et al.
Published: (2025)

INMS: Memory Sharing for Large Language Model based Agents
by: Gao, Hang, et al.
Published: (2024)

Capybara-OMNI: An Efficient Paradigm for Building Omni-Modal Language Models
by: Ji, Xingguang, et al.
Published: (2025)

Efficient Pretraining Length Scaling
by: Wu, Bohong, et al.
Published: (2025)

SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models
by: Liu, Han, et al.
Published: (2026)

A Systematic Survey of Semantic Role Labeling in the Era of Pretrained Language Models
by: Chen, Huiyao, et al.
Published: (2025)

Benchmarking Gender and Political Bias in Large Language Models
by: Yang, Jinrui, et al.
Published: (2025)

A Survey on Efficient Large Language Model Training: From Data-centric Perspectives
by: Luo, Junyu, et al.
Published: (2025)

DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization
by: Deng, Hexuan, et al.
Published: (2024)

Adapting Pretrained Language Models for Citation Classification via Self-Supervised Contrastive Learning
by: Li, Tong, et al.
Published: (2025)

Semantic Reformulation Entropy for Robust Hallucination Detection in QA Tasks
by: Tong, Chaodong, et al.
Published: (2025)

RECKON: Large-scale Reference-based Efficient Knowledge Evaluation for Large Language Model
by: Zhang, Lin, et al.
Published: (2025)

PretrainRL: Alleviating Factuality Hallucination of Large Language Models at the Beginning
by: Liu, Langming, et al.
Published: (2026)

Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining
by: Guo, Ping, et al.
Published: (2025)

Adaptive Draft-Verification for Efficient Large Language Model Decoding
by: Liu, Xukun, et al.
Published: (2024)

FlashDecoding++: Faster Large Language Model Inference on GPUs
by: Hong, Ke, et al.
Published: (2023)

MASS: Mathematical Data Selection via Skill Graphs for Pretraining Large Language Models
by: Li, Jiazheng, et al.
Published: (2025)

ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models
by: Liu, Jing, et al.
Published: (2024)

The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis
by: Yang, Chen, et al.
Published: (2024)

Fine-tuning can Help Detect Pretraining Data from Large Language Models
by: Zhang, Hengxiang, et al.
Published: (2024)