Cover Image

Democratizing AI: Open-source Scalable LLM Training on GPU-based Supercomputers

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Singh, Siddharth, Singhania, Prajwal, Ranjan, Aditya, Kirchenbauer, John, Geiping, Jonas, Wen, Yuxin, Jain, Neel, Hans, Abhimanyu, Shu, Manli, Tomar, Aditya, Goldstein, Tom, Bhatele, Abhinav
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMs
by: Hans, Abhimanyu, et al.
Published: (2024)

A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUs
by: Singh, Siddharth, et al.
Published: (2023)

Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach
by: Geiping, Jonas, et al.
Published: (2025)

Speculating Experts Accelerates Inference for Mixture-of-Experts
by: Madan, Vivan, et al.
Published: (2026)

Power Law Guided Dynamic Sifting for Efficient Attention
by: Koley, Nirav, et al.
Published: (2025)

Loki: Low-rank Keys for Efficient Sparse Attention
by: Singhania, Prajwal, et al.
Published: (2024)

Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Training
by: Ranjan, Aditya K., et al.
Published: (2025)

Transformers Can Do Arithmetic with the Right Embeddings
by: McLeish, Sean, et al.
Published: (2024)

Coercing LLMs to do and reveal (almost) anything
by: Geiping, Jonas, et al.
Published: (2024)

A Watermark for Large Language Models
by: Kirchenbauer, John, et al.
Published: (2023)

On the Reliability of Watermarks for Large Language Models
by: Kirchenbauer, John, et al.
Published: (2023)

Optimizing Agentic Language Model Inference via Speculative Tool Calls
by: Nichols, Daniel, et al.
Published: (2025)

Understanding and Improving Communication Performance in Multi-node LLM Inference
by: Singhania, Prajwal, et al.
Published: (2025)

Exploiting Sparsity for Long Context Inference: Million Token Contexts on Commodity GPUs
by: Synk, Ryan, et al.
Published: (2025)

Gemstones: A Model Suite for Multi-Faceted Scaling Laws
by: McLeish, Sean, et al.
Published: (2025)

GenQA: Generating Millions of Instructions from a Handful of Prompts
by: Chen, Jiuhai, et al.
Published: (2024)

Pipit: Scripting the analysis of parallel execution traces
by: Bhatele, Abhinav, et al.
Published: (2023)

Automated Programmatic Performance Analysis of Parallel Programs
by: Cankur, Onur, et al.
Published: (2024)

LMD3: Language Model Data Density Dependence
by: Kirchenbauer, John, et al.
Published: (2024)

Multi-Token Prediction via Self-Distillation
by: Kirchenbauer, John, et al.
Published: (2026)

Zero-Shot Vision Encoder Grafting via LLM Surrogates
by: Yue, Kaiyu, et al.
Published: (2025)

Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Training
by: Wei, Cunyang, et al.
Published: (2026)

The Big Send-off: Scalable and Performant Collectives for Deep Learning
by: Singh, Siddharth, et al.
Published: (2025)

FictionalQA: A Dataset for Studying Memorization and Knowledge Acquisition
by: Kirchenbauer, John, et al.
Published: (2025)

Privacy Backdoors: Enhancing Membership Inference through Poisoning Pre-trained Models
by: Wen, Yuxin, et al.
Published: (2024)

BharatBBQ: A Multilingual Bias Benchmark for Question Answering in the Indian Context
by: Tomar, Aditya, et al.
Published: (2025)

Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
by: Hans, Abhimanyu, et al.
Published: (2024)

Characterizing Production GPU Workloads using System-wide Telemetry Data
by: Cankur, Onur, et al.
Published: (2025)

HPC-Coder-V2: Studying Code LLMs Across Low-Resource Parallel Languages
by: Chaturvedi, Aman, et al.
Published: (2024)

What do we learn from inverting CLIP models?
by: Kazemi, Hamid, et al.
Published: (2024)

Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale
by: Zhao, Dan, et al.
Published: (2024)

Integrating Performance Tools in Model Reasoning for GPU Kernel Optimization
by: Nichols, Daniel, et al.
Published: (2025)

Measuring Style Similarity in Diffusion Models
by: Somepalli, Gowthami, et al.
Published: (2024)

When Can You Get Away with Low Memory Adam?
by: Kalra, Dayal Singh, et al.
Published: (2025)

KEET: Explaining Performance of GPU Kernels Using LLM Agents
by: Davis, Joshua H., et al.
Published: (2026)

Exploring GPU-to-GPU Communication: Insights into Supercomputer Interconnects
by: De Sensi, Daniele, et al.
Published: (2024)

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence
by: McLeish, Sean, et al.
Published: (2025)

Analytics of Longitudinal System Monitoring Data for Performance Prediction
by: Costello, Ian J., et al.
Published: (2020)

Refusal Tokens: A Simple Way to Calibrate Refusals in Large Language Models
by: Jain, Neel, et al.
Published: (2024)

Mathematics Isn't Culture-Free: Probing Cultural Gaps via Entity and Scenario Perturbations
by: Tomar, Aditya, et al.
Published: (2025)