Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918452820705280 |
|---|---|
| author | Wang, Dong Li, Yang Ni, Ansong Yeh, Ching-Feng Emad, Youssef Lei, Xinjie Robbins, Liam Padthe, Karthik Xu, Hu Li, Xian Celikyilmaz, Asli Raghavendra, Ramya Huang, Lifei Wu, Carole-Jean Li, Shang-Wen |
| author_facet | Wang, Dong Li, Yang Ni, Ansong Yeh, Ching-Feng Emad, Youssef Lei, Xinjie Robbins, Liam Padthe, Karthik Xu, Hu Li, Xian Celikyilmaz, Asli Raghavendra, Ramya Huang, Lifei Wu, Carole-Jean Li, Shang-Wen |
| contents | Synthetic data has become increasingly important for training large language models, especially when real data is scarce, expensive, or privacy-sensitive. Many such generation tasks require coordinated multi-agent workflows, where specialized agents collaborate to produce data that is higher quality, more diverse, and structurally richer. However, existing frameworks for multi-agent synthesis often depend on a centralized orchestrator, creating scalability bottlenecks, or are hardcoded for specific domains, limiting flexibility. We present \textbf{Matrix}, a decentralized framework that represents both control and data flow as serialized messages passed through distributed queues. This peer-to-peer design eliminates the central orchestrator. Each task progresses independently through lightweight agents, while compute-intensive operations, such as LLM inference or containerized environments, are handled by distributed services. Built on Ray, Matrix scales to tens of thousands of concurrent agentic workflows and provides a modular, configurable design that enables easy adaptation to a wide range of data generation workflows. We evaluate Matrix across diverse synthesis scenarios, such as multi-agent collaborative dialogue, web-based reasoning data extraction, and tool-use trajectory generation in customer service environments. In all cases, Matrix achieves $2$--$15\times$ higher data generation throughput under identical hardware resources, without compromising output quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_21686 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework Wang, Dong Li, Yang Ni, Ansong Yeh, Ching-Feng Emad, Youssef Lei, Xinjie Robbins, Liam Padthe, Karthik Xu, Hu Li, Xian Celikyilmaz, Asli Raghavendra, Ramya Huang, Lifei Wu, Carole-Jean Li, Shang-Wen Computation and Language Artificial Intelligence Machine Learning Synthetic data has become increasingly important for training large language models, especially when real data is scarce, expensive, or privacy-sensitive. Many such generation tasks require coordinated multi-agent workflows, where specialized agents collaborate to produce data that is higher quality, more diverse, and structurally richer. However, existing frameworks for multi-agent synthesis often depend on a centralized orchestrator, creating scalability bottlenecks, or are hardcoded for specific domains, limiting flexibility. We present \textbf{Matrix}, a decentralized framework that represents both control and data flow as serialized messages passed through distributed queues. This peer-to-peer design eliminates the central orchestrator. Each task progresses independently through lightweight agents, while compute-intensive operations, such as LLM inference or containerized environments, are handled by distributed services. Built on Ray, Matrix scales to tens of thousands of concurrent agentic workflows and provides a modular, configurable design that enables easy adaptation to a wide range of data generation workflows. We evaluate Matrix across diverse synthesis scenarios, such as multi-agent collaborative dialogue, web-based reasoning data extraction, and tool-use trajectory generation in customer service environments. In all cases, Matrix achieves $2$--$15\times$ higher data generation throughput under identical hardware resources, without compromising output quality. |
| title | Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Framework |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2511.21686 |