Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.18784 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914168650596352 |
|---|---|
| author | Yu, Xinghang Yu, Ce Shao, Zeguang Yang, Bin |
| author_facet | Yu, Xinghang Yu, Ce Shao, Zeguang Yang, Bin |
| contents | With the exponential growth of time-domain surveys, the volume of light curves has increased rapidly. However, many survey projects, such as Gaia, still rely on offline batch-processing workflows in which data are calibrated, merged, and released only after an observing phase is completed. This latency delays scientific analysis and causes many high-value transient events to be buried in archival data, missing the window for timely follow-up. While existing alert brokers handle heterogeneous data streams, it remains difficult to deploy a unified framework that combines high-performance incremental storage with real-time classification on local infrastructure. To address this challenge, we propose TDLight, a scalable system that adapts the time-series database TDengine (a high-performance IoT database) for astronomical data using a one-table-per-source schema. This architecture supports high-throughput ingestion, achieving 954,000 rows s^-1 for archived data and 541,000 rows s^-1 for incremental streams, while Hierarchical Equal Area isoLatitude Pixelization (HEALPix) indexing enables efficient cone-search queries. Building on this storage layer, we integrate the pre-trained hierarchical Random Forest classifier from the LEAVES framework to construct an incremental classification pipeline. Using the LEAVES dataset, we simulate data accumulation and evaluate a trigger-based strategy that performs early classification at specific observational milestones. In addition, by monitoring the evolution of classification probabilities, the system identifies "high-value candidates" -- sources that show high early confidence but later undergo significant label shifts. TDLight is released as an open-source Dockerized environment, providing a deployable infrastructure for next-generation time-domain surveys. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_18784 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TDLight: A Framework for Incremental Light Curve Management and Smart Classification Yu, Xinghang Yu, Ce Shao, Zeguang Yang, Bin Instrumentation and Methods for Astrophysics With the exponential growth of time-domain surveys, the volume of light curves has increased rapidly. However, many survey projects, such as Gaia, still rely on offline batch-processing workflows in which data are calibrated, merged, and released only after an observing phase is completed. This latency delays scientific analysis and causes many high-value transient events to be buried in archival data, missing the window for timely follow-up. While existing alert brokers handle heterogeneous data streams, it remains difficult to deploy a unified framework that combines high-performance incremental storage with real-time classification on local infrastructure. To address this challenge, we propose TDLight, a scalable system that adapts the time-series database TDengine (a high-performance IoT database) for astronomical data using a one-table-per-source schema. This architecture supports high-throughput ingestion, achieving 954,000 rows s^-1 for archived data and 541,000 rows s^-1 for incremental streams, while Hierarchical Equal Area isoLatitude Pixelization (HEALPix) indexing enables efficient cone-search queries. Building on this storage layer, we integrate the pre-trained hierarchical Random Forest classifier from the LEAVES framework to construct an incremental classification pipeline. Using the LEAVES dataset, we simulate data accumulation and evaluate a trigger-based strategy that performs early classification at specific observational milestones. In addition, by monitoring the evolution of classification probabilities, the system identifies "high-value candidates" -- sources that show high early confidence but later undergo significant label shifts. TDLight is released as an open-source Dockerized environment, providing a deployable infrastructure for next-generation time-domain surveys. |
| title | TDLight: A Framework for Incremental Light Curve Management and Smart Classification |
| topic | Instrumentation and Methods for Astrophysics |
| url | https://arxiv.org/abs/2511.18784 |