FalconFS: Distributed File System for Large-Scale Deep Learning Pipeline
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917188767580160 |
|---|---|
| author | Xu, Jingwei Kang, Junbin Dong, Mingkai Liu, Mingyu Zhang, Lu Guo, Shaohong Qiu, Ziyan You, Mingzhen Tian, Ziyi Yu, Anqi Ding, Tianhong Hu, Xinwei Chen, Haibo |
| author_facet | Xu, Jingwei Kang, Junbin Dong, Mingkai Liu, Mingyu Zhang, Lu Guo, Shaohong Qiu, Ziyan You, Mingzhen Tian, Ziyi Yu, Anqi Ding, Tianhong Hu, Xinwei Chen, Haibo |
| contents | Client-side metadata caching has long been considered an effective method for accelerating metadata operations in distributed file systems (DFSs). However, we have found that client-side state (e.g., caching) is not only ineffective but also consumes valuable memory resources in the deep learning pipelines. We thus propose FalconFS, a DFS optimized for deep learning pipelines with the stateless-client architecture. Specifically, instead of performing client-side path resolution and caching, FalconFS efficiently resolves paths on the server side using hybrid metadata indexing and lazy namespace replication. FalconFS also boosts server concurrency with concurrent request merging and provides easy deployment with VFS shortcut. Evaluations against CephFS and Lustre show that FalconFS achieves up to 5.72$\times$ throughput for small file read/write and up to 12.81$\times$ throughput for deep learning model training. FalconFS has been running in Huawei autonomous driving system's production environment with 10,000 NPUs for one year and has been open-sourced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_10367 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FalconFS: Distributed File System for Large-Scale Deep Learning Pipeline Xu, Jingwei Kang, Junbin Dong, Mingkai Liu, Mingyu Zhang, Lu Guo, Shaohong Qiu, Ziyan You, Mingzhen Tian, Ziyi Yu, Anqi Ding, Tianhong Hu, Xinwei Chen, Haibo Distributed, Parallel, and Cluster Computing Performance Client-side metadata caching has long been considered an effective method for accelerating metadata operations in distributed file systems (DFSs). However, we have found that client-side state (e.g., caching) is not only ineffective but also consumes valuable memory resources in the deep learning pipelines. We thus propose FalconFS, a DFS optimized for deep learning pipelines with the stateless-client architecture. Specifically, instead of performing client-side path resolution and caching, FalconFS efficiently resolves paths on the server side using hybrid metadata indexing and lazy namespace replication. FalconFS also boosts server concurrency with concurrent request merging and provides easy deployment with VFS shortcut. Evaluations against CephFS and Lustre show that FalconFS achieves up to 5.72$\times$ throughput for small file read/write and up to 12.81$\times$ throughput for deep learning model training. FalconFS has been running in Huawei autonomous driving system's production environment with 10,000 NPUs for one year and has been open-sourced. |
| title | FalconFS: Distributed File System for Large-Scale Deep Learning Pipeline |
| topic | Distributed, Parallel, and Cluster Computing Performance |
| url | https://arxiv.org/abs/2507.10367 |