Modular Architecture for High-Performance and Low Overhead Data Transfers
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911255197908992 |
|---|---|
| author | Swargo, Rasman Mubtasim Arslan, Engin Arifuzzaman, Md |
| author_facet | Swargo, Rasman Mubtasim Arslan, Engin Arifuzzaman, Md |
| contents | High-performance applications necessitate rapid and dependable transfer of massive datasets across geographically dispersed locations. Traditional file transfer tools often suffer from resource underutilization and instability because of fixed configurations or monolithic optimization methods. We propose AutoMDT, a novel modular data transfer architecture that employs a deep reinforcement learning based agent to simultaneously optimize concurrency levels for read, network, and write operations. Our solution incorporates a lightweight network-system simulator, enabling offline training of a Proximal Policy Optimization (PPO) agent in approximately 45 minutes on average, thereby overcoming the impracticality of lengthy online training in production networks. AutoMDT's modular design decouples I/O and network tasks, allowing the agent to capture complex buffer dynamics precisely and to adapt quickly to changing system and network conditions. Evaluations on production-grade testbeds show that AutoMDT achieves up to 8x faster convergence and a 68% reduction in transfer completion times compared with state-of-the-art solutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_05546 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Modular Architecture for High-Performance and Low Overhead Data Transfers Swargo, Rasman Mubtasim Arslan, Engin Arifuzzaman, Md Distributed, Parallel, and Cluster Computing High-performance applications necessitate rapid and dependable transfer of massive datasets across geographically dispersed locations. Traditional file transfer tools often suffer from resource underutilization and instability because of fixed configurations or monolithic optimization methods. We propose AutoMDT, a novel modular data transfer architecture that employs a deep reinforcement learning based agent to simultaneously optimize concurrency levels for read, network, and write operations. Our solution incorporates a lightweight network-system simulator, enabling offline training of a Proximal Policy Optimization (PPO) agent in approximately 45 minutes on average, thereby overcoming the impracticality of lengthy online training in production networks. AutoMDT's modular design decouples I/O and network tasks, allowing the agent to capture complex buffer dynamics precisely and to adapt quickly to changing system and network conditions. Evaluations on production-grade testbeds show that AutoMDT achieves up to 8x faster convergence and a 68% reduction in transfer completion times compared with state-of-the-art solutions. |
| title | Modular Architecture for High-Performance and Low Overhead Data Transfers |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2508.05546 |