Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917401135677440 |
|---|---|
| author | Pan, Haochen Chard, Ryan Oh, Song Young Gonthier, Maxime Hayot-Sasson, Valérie Lentner, Geoffrey Bottigliero, Joe Ananthakrishnan, Rachana Chard, Kyle Foster, Ian |
| author_facet | Pan, Haochen Chard, Ryan Oh, Song Young Gonthier, Maxime Hayot-Sasson, Valérie Lentner, Geoffrey Bottigliero, Joe Ananthakrishnan, Rachana Chard, Kyle Foster, Ian |
| contents | Modern HPC file systems can contain billions of files and hundreds of petabytes of data, making even simple questions increasingly intractable to answer. Traditional file system utilities such as find and du fail to scale to these sizes. While external indexing tools like GUFI and Brindexer improve query performance, they remain batch-oriented and unsuitable for heterogeneous, rapidly evolving environments.
We present Icicle, a scalable framework for continuous file system metadata indexing and monitoring. Icicle maintains a unified, up-to-date, and queryable view of file system state while supporting both periodic snapshot-based ingestion for bulk metadata updates and event-based ingestion for real-time synchronization from production systems such as Lustre and IBM Storage Scale. Built on Apache Kafka and Apache Flink, Icicle provides high-throughput, fault-tolerant, and horizontally scalable ingestion of metadata events into two complementary search indexes, enabling both individual file discovery and aggregate summary statistics by user, group, and directory.
This architecture enables efficient support for both coarse-grained administrative queries and interactive analytics over billions of objects. Our experimental evaluation on production-scale HPC datasets demonstrates order-of-magnitude throughput improvements over existing monitoring and indexing approaches, with tunable options for balancing consistency, latency, and metadata freshness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_10295 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems Pan, Haochen Chard, Ryan Oh, Song Young Gonthier, Maxime Hayot-Sasson, Valérie Lentner, Geoffrey Bottigliero, Joe Ananthakrishnan, Rachana Chard, Kyle Foster, Ian Distributed, Parallel, and Cluster Computing Computational Engineering, Finance, and Science Modern HPC file systems can contain billions of files and hundreds of petabytes of data, making even simple questions increasingly intractable to answer. Traditional file system utilities such as find and du fail to scale to these sizes. While external indexing tools like GUFI and Brindexer improve query performance, they remain batch-oriented and unsuitable for heterogeneous, rapidly evolving environments. We present Icicle, a scalable framework for continuous file system metadata indexing and monitoring. Icicle maintains a unified, up-to-date, and queryable view of file system state while supporting both periodic snapshot-based ingestion for bulk metadata updates and event-based ingestion for real-time synchronization from production systems such as Lustre and IBM Storage Scale. Built on Apache Kafka and Apache Flink, Icicle provides high-throughput, fault-tolerant, and horizontally scalable ingestion of metadata events into two complementary search indexes, enabling both individual file discovery and aggregate summary statistics by user, group, and directory. This architecture enables efficient support for both coarse-grained administrative queries and interactive analytics over billions of objects. Our experimental evaluation on production-scale HPC datasets demonstrates order-of-magnitude throughput improvements over existing monitoring and indexing approaches, with tunable options for balancing consistency, latency, and metadata freshness. |
| title | Icicle: Scalable Metadata Indexing and Real-Time Monitoring for HPC File Systems |
| topic | Distributed, Parallel, and Cluster Computing Computational Engineering, Finance, and Science |
| url | https://arxiv.org/abs/2604.10295 |