Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2510.04012 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911428505501696 |
|---|---|
| author | Rogers, David Mariani, Valerio Wang, Cong Coffee, Ryan Kroeger, Wilko Shankar, Murali Schwander, Hans Thorsten Beck, Tom Poitevin, Frédéric Thayer, Jana |
| author_facet | Rogers, David Mariani, Valerio Wang, Cong Coffee, Ryan Kroeger, Wilko Shankar, Murali Schwander, Hans Thorsten Beck, Tom Poitevin, Frédéric Thayer, Jana |
| contents | We describe a new end-to-end experimental data streaming framework designed from the ground up to support new types of applications -- AI training, extremely high-rate X-ray time-of-flight analysis, crystal structure determination with distributed processing, and custom data science applications and visualizers yet to be created. Throughout, we use design choices merging cloud microservices with traditional HPC batch execution models for security and flexibility. This project makes a unique contribution to the DOE Integrated Research Infrastructure (IRI) landscape. By creating a flexible, API-driven data request service, we address a significant need for high-speed data streaming sources for the X-ray science data analysis community. With the combination of data request API, mutual authentication web security framework, job queue system, high-rate data buffer, and complementary nature to facility infrastructure, the LCLStreamer framework has prototyped and implemented several new paradigms critical for future generation experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_04012 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The LCLStream Ecosystem for Multi-Institutional Dataset Exploration Rogers, David Mariani, Valerio Wang, Cong Coffee, Ryan Kroeger, Wilko Shankar, Murali Schwander, Hans Thorsten Beck, Tom Poitevin, Frédéric Thayer, Jana Information Retrieval Instrumentation and Detectors We describe a new end-to-end experimental data streaming framework designed from the ground up to support new types of applications -- AI training, extremely high-rate X-ray time-of-flight analysis, crystal structure determination with distributed processing, and custom data science applications and visualizers yet to be created. Throughout, we use design choices merging cloud microservices with traditional HPC batch execution models for security and flexibility. This project makes a unique contribution to the DOE Integrated Research Infrastructure (IRI) landscape. By creating a flexible, API-driven data request service, we address a significant need for high-speed data streaming sources for the X-ray science data analysis community. With the combination of data request API, mutual authentication web security framework, job queue system, high-rate data buffer, and complementary nature to facility infrastructure, the LCLStreamer framework has prototyped and implemented several new paradigms critical for future generation experiments. |
| title | The LCLStream Ecosystem for Multi-Institutional Dataset Exploration |
| topic | Information Retrieval Instrumentation and Detectors |
| url | https://arxiv.org/abs/2510.04012 |