XForecast: Evaluating Natural Language Explanations for Time Series Forecasting
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910657803190272 |
|---|---|
| author | Aksu, Taha Liu, Chenghao Saha, Amrita Tan, Sarah Xiong, Caiming Sahoo, Doyen |
| author_facet | Aksu, Taha Liu, Chenghao Saha, Amrita Tan, Sarah Xiong, Caiming Sahoo, Doyen |
| contents | Time series forecasting aids decision-making, especially for stakeholders who rely on accurate predictions, making it very important to understand and explain these models to ensure informed decisions. Traditional explainable AI (XAI) methods, which underline feature or temporal importance, often require expert knowledge. In contrast, natural language explanations (NLEs) are more accessible to laypeople. However, evaluating forecast NLEs is difficult due to the complex causal relationships in time series data. To address this, we introduce two new performance metrics based on simulatability, assessing how well a human surrogate can predict model forecasts using the explanations. Experiments show these metrics differentiate good from poor explanations and align with human judgments. Utilizing these metrics, we further evaluate the ability of state-of-the-art large language models (LLMs) to generate explanations for time series data, finding that numerical reasoning, rather than model size, is the main factor influencing explanation quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_14180 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | XForecast: Evaluating Natural Language Explanations for Time Series Forecasting Aksu, Taha Liu, Chenghao Saha, Amrita Tan, Sarah Xiong, Caiming Sahoo, Doyen Computation and Language Time series forecasting aids decision-making, especially for stakeholders who rely on accurate predictions, making it very important to understand and explain these models to ensure informed decisions. Traditional explainable AI (XAI) methods, which underline feature or temporal importance, often require expert knowledge. In contrast, natural language explanations (NLEs) are more accessible to laypeople. However, evaluating forecast NLEs is difficult due to the complex causal relationships in time series data. To address this, we introduce two new performance metrics based on simulatability, assessing how well a human surrogate can predict model forecasts using the explanations. Experiments show these metrics differentiate good from poor explanations and align with human judgments. Utilizing these metrics, we further evaluate the ability of state-of-the-art large language models (LLMs) to generate explanations for time series data, finding that numerical reasoning, rather than model size, is the main factor influencing explanation quality. |
| title | XForecast: Evaluating Natural Language Explanations for Time Series Forecasting |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.14180 |