Cover Image

The Future of Large Language Model Pre-training is Federated

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Sani, Lorenzo, Iacob, Alex, Cao, Zeyu, Marino, Bill, Gao, Yan, Paulik, Tomas, Zhao, Wanru, Shen, William F., Aleksandrov, Preslav, Qiu, Xinchi, Lane, Nicholas D.
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Photon: Federated LLM Pre-Training
by: Sani, Lorenzo, et al.
Published: (2024)

Worldwide Federated Training of Language Models
by: Iacob, Alex, et al.
Published: (2024)

Pollen: High-throughput Federated Learning Simulation via Resource-Aware Client Placement
by: Sani, Lorenzo, et al.
Published: (2023)

Cascadia: An Efficient Cascade Serving System for Large Language Models
by: Jiang, Youhe, et al.
Published: (2025)

Enhancing Data Quality in Federated Fine-Tuning of Foundation Models
by: Zhao, Wanru, et al.
Published: (2024)

FLea: Addressing Data Scarcity and Label Skew in Federated Learning via Privacy-preserving Feature Augmentation
by: Xia, Tong, et al.
Published: (2024)

Gradient-less Federated Gradient Boosting Trees with Learnable Learning Rates
by: Ma, Chenyang, et al.
Published: (2023)

Vertical Federated Learning Hybrid Local Pre-training
by: Li, Wenguo, et al.
Published: (2024)

Straggler Tolerant and Resilient DL Training on Homogeneous GPUs
by: Zhang, Zeyu, et al.
Published: (2025)

FLea: Addressing Data Scarcity and Label Skew in Federated Learning via Privacy-preserving Feature Augmentation
by: Xia, Tong, et al.
Published: (2023)

GPT-FL: Generative Pre-trained Model-Assisted Federated Learning
by: Zhang, Tuo, et al.
Published: (2023)

An Upload-Efficient Scheme for Transferring Knowledge From a Server-Side Pre-trained Generator to Clients in Heterogeneous Federated Learning
by: Zhang, Jianqing, et al.
Published: (2024)

Optimization of Federated Learning's Client Selection for Non-IID Data Based on Grey Relational Analysis
by: Chen, Shuaijun, et al.
Published: (2023)

GeoFF: Federated Serverless Workflows with Data Pre-Fetching
by: Carl, Natalie, et al.
Published: (2024)

FLMarket: Enabling Privacy-preserved Pre-training Data Pricing for Federated Learning
by: Wen, Zhenyu, et al.
Published: (2024)

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism
by: Zhang, Zheng, et al.
Published: (2025)

Federated Unlearning in Edge Networks: A Survey of Fundamentals, Challenges, Practical Applications and Future Directions
by: Ng, Jer Shyuan, et al.
Published: (2026)

Split Federated Learning Empowered Vehicular Edge Intelligence: Concept, Adaptive Design and Future Directions
by: Qiang, Xianke, et al.
Published: (2024)

Floe: Federated Specialization for Real-Time LLM-SLM Inference
by: Tian, Chunlin, et al.
Published: (2026)

DART: A Solution for Decentralized Federated Learning Model Robustness Analysis
by: Feng, Chao, et al.
Published: (2024)

FedHC: A Hierarchical Clustered Federated Learning Framework for Satellite Networks
by: Liu, Zhuocheng, et al.
Published: (2025)

Advancing AI-assisted Hardware Design with Hierarchical Decentralized Training and Personalized Inference-Time Optimization
by: Chen, Hao Mark, et al.
Published: (2025)

FDC: Fast KV Dimensionality Compression for Efficient LLM Inference
by: Zhang, Zeyu, et al.
Published: (2024)

PecSched: Preemptive and Efficient Cluster Scheduling for LLM Inference
by: Zhang, Zeyu, et al.
Published: (2024)

CacheFL: Privacy-Preserving and Efficient Federated Cache Model Fine-Tuning for Vision-Language Models
by: Yi, Mengjun, et al.
Published: (2025)

EdgeShard: Efficient LLM Inference via Collaborative Edge Computing
by: Zhang, Mingjin, et al.
Published: (2024)

Fog enabled distributed training architecture for federated learning
by: Kumar, Aditya, et al.
Published: (2024)

Initialization Matters: Unraveling the Impact of Pre-Training on Federated Learning
by: Jhunjhunwala, Divyansh, et al.
Published: (2025)

Lattica: A Decentralized Cross-NAT Communication Framework for Scalable AI Inference and Training
by: Yang, Ween, et al.
Published: (2025)

Simulating LLM training workloads for heterogeneous compute and network infrastructure
by: Kumar, Sumit, et al.
Published: (2025)

Sustainable Edge Computing: Challenges and Future Directions
by: Arroba, Patricia, et al.
Published: (2023)

A Preliminary Study on Accelerating Simulation Optimization with GPU Implementation
by: He, Jinghai, et al.
Published: (2024)

Augur: Pre-Execution Energy Prediction for Workflow Tasks in Heterogeneous Clusters
by: West, Kathleen, et al.
Published: (2026)

Beyond Pre-Training: The Full Lifecycle of Foundation Models on HPC Systems
by: Conciatore, Dino, et al.
Published: (2026)

Federated Learning for Terahertz Wireless Communication
by: Baydas, O. Tansel, et al.
Published: (2025)

Incentive-Based Federated Learning: Architectural Elements and Future Directions
by: Kaluannakkage, Chanuka A. S. Hewa, et al.
Published: (2025)

Back to the Future: Rethinking Endorsement in Order-Execute Blockchains
by: Huang, Rongji, et al.
Published: (2026)

FedORGP: Guiding Heterogeneous Federated Learning with Orthogonality Regularization on Global Prototypes
by: Guo, Fucheng, et al.
Published: (2025)

Unleashing Scalable Context Parallelism for Foundation Models Pre-Training via FCP
by: Zhao, Yilong, et al.
Published: (2026)

Open Science Data Federation -- operation and monitoring
by: Andrijauskas, Fabio, et al.
Published: (2026)