Cover Image

Data-Juicer 2.0: Cloud-Scale Adaptive Data Processing for and with Foundation Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Chen, Daoyuan, Huang, Yilun, Pan, Xuchen, Jiang, Nana, Wang, Haibin, Zhang, Yilei, Ge, Ce, Chen, Yushuo, Zhang, Wenhao, Ma, Zhijian, Huang, Jun, Lin, Wei, Li, Yaliang, Ding, Bolin, Zhou, Jingren
Format:	Preprint
Published:	2024
Subjects:	Distributed, Parallel, and Cluster Computing Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Trinity-RFT: A General-Purpose and Unified Framework for Reinforcement Fine-Tuning of Large Language Models
by: Pan, Xuchen, et al.
Published: (2025)

Data-Juicer Sandbox: A Feedback-Driven Suite for Multimodal Data-Model Co-development
by: Chen, Daoyuan, et al.
Published: (2024)

EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelism
by: Chen, Yanxi, et al.
Published: (2023)

Federated Full-Parameter Tuning of Billion-Sized Language Models with Communication Cost under 18 Kilobytes
by: Qin, Zhen, et al.
Published: (2023)

Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing
by: Lyu, Chenghao, et al.
Published: (2022)

Exploring Selective Layer Fine-Tuning in Federated Learning
by: Sun, Yuchang, et al.
Published: (2024)

FuxiShuffle: An Adaptive and Resilient Shuffle Service for Distributed Data Processing on Alibaba Cloud
by: Lin, Yuhao, et al.
Published: (2026)

Adaptive K-PackCache: Cost-Centric Data Caching in Cloud
by: Sarkar, Suvarthi, et al.
Published: (2025)

Improving LoRA in Privacy-preserving Federated Learning
by: Sun, Youbang, et al.
Published: (2024)

On-the-fly Communication-and-Computing to Enable Representation Learning for Distributed Point Clouds
by: Chen, Xu, et al.
Published: (2024)

Crossword: Adaptive Consensus for Dynamic Data-Heavy Workloads
by: Hu, Guanzhou, et al.
Published: (2025)

AgentFlow: Resilient Adaptive Cloud-Edge Framework for Multi-Agent Coordination
by: Chen, Ching Han, et al.
Published: (2025)

Truffle: Efficient Data Passing for Data-Intensive Serverless Workflows in the Edge-Cloud Continuum
by: Marcelino, Cynthia, et al.
Published: (2024)

FedBiOT: LLM Local Fine-tuning in Federated Learning without Full Model
by: Wu, Feijie, et al.
Published: (2024)

MegaScale-Data: Scaling Dataloader for Multisource Large Foundation Model Training
by: Zhao, Juntao, et al.
Published: (2025)

Optimization Opportunities for Cloud-Based Data Pipeline Infrastructures
by: Jablonski, Johannes, et al.
Published: (2026)

UMDAM: A Unified Data Layout and DRAM Address Mapping for Heterogenous NPU-PIM
by: Huang, Hai
Published: (2025)

Seer: Proactive Revenue-Aware Scheduling for Live Streaming Services in Crowdsourced Cloud-Edge Platforms
by: Huang, Shaoyuan, et al.
Published: (2024)

Performance Cost Tradeoffs in Intelligent Load Balancing for Multi Data Center Cloud Systems: From Static Policies to Adaptive Resource Distribution
by: Najafabadi, Saeid Aghasoleymani, et al.
Published: (2025)

DHLink: A Microservice Platform supporting Rapid Application Development and Secure Real-time Data Sharing in Digital Health
by: Li, Wenhao, et al.
Published: (2021)

Towards Cloud Efficiency with Large-scale Workload Characterization
by: Parayil, Anjaly, et al.
Published: (2024)

Environmentally-Conscious Cloud Orchestration Considering Geo-Distributed Data Centers
by: Attenni, Giulio, et al.
Published: (2025)

LSKV: A Confidential Distributed Datastore to Protect Critical Data in the Cloud
by: Jeffery, Andrew, et al.
Published: (2024)

Nimbus: A Unified Embodied Synthetic Data Generation Framework
by: He, Zeyu, et al.
Published: (2026)

GPZ: GPU-Accelerated Lossy Compressor for Particle Data
by: Li, Ruoyu, et al.
Published: (2025)

Self-healing Nodes with Adaptive Data-Sharding
by: Thakur, Ayush, et al.
Published: (2024)

A Cloud in the Sky: Geo-Aware On-board Data Services for LEO Satellites
by: Sandholm, Thomas, et al.
Published: (2024)

A New Perspective of Graph Data and A Generic and Efficient Method for Large Scale Graph Data Traversal
by: Zhang, Chenglong
Published: (2020)

Towards an Adaptive Runtime System for Cloud-Native HPC
by: Bhosale, Aditya, et al.
Published: (2026)

Flora: Efficient Cloud Resource Selection for Big Data Processing via Job Classification
by: Will, Jonathan, et al.
Published: (2025)

FedAC: An Adaptive Clustered Federated Learning Framework for Heterogeneous Data
by: Zhang, Yuxin, et al.
Published: (2024)

Federated Learning Using Coupled Tensor Train Decomposition
by: Zhang, Xiangtao, et al.
Published: (2024)

Optimization and Application of Cloud-based Deep Learning Architecture for Multi-Source Data Prediction
by: Zhang, Yang, et al.
Published: (2024)

PolarStore: High-Performance Data Compression for Large-Scale Cloud-Native Databases
by: Hu, Qingda, et al.
Published: (2025)

Grounded in Reality: Learning and Deploying Proactive LLM from Offline Logs
by: Wei, Fei, et al.
Published: (2025)

Parallel Seismic Data Processing Performance with Cloud-based Storage
by: Mohapatra, Sasmita, et al.
Published: (2025)

Push Down Optimization for Distributed Multi Cloud Data Integration
by: Kodali, Ravi Kiran, et al.
Published: (2026)

Radar DataTree: A FAIR and Cloud-Native Framework for Scalable Weather Radar Archives
by: Ladino-Rincon, Alfonso, et al.
Published: (2025)

SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
by: Jaiswal, Shashwat, et al.
Published: (2025)

Towards Designing an Energy Aware Data Replication Strategy for Cloud Systems Using Reinforcement Learning
by: Najjar, Amir, et al.
Published: (2025)