JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918406279659520 |
|---|---|
| author | Wang, Yuhui Yang, Zhixiong Zhang, Ming Dou, Shihan Xi, Zhiheng Zhou, Enyu Jin, Senjie Shen, Yujiong Zhu, Dingwei Dong, Yi Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Wang, Yuhui Yang, Zhixiong Zhang, Ming Dou, Shihan Xi, Zhiheng Zhou, Enyu Jin, Senjie Shen, Yujiong Zhu, Dingwei Dong, Yi Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains $3130$ entries and $40.75$ turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_22978 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees Wang, Yuhui Yang, Zhixiong Zhang, Ming Dou, Shihan Xi, Zhiheng Zhou, Enyu Jin, Senjie Shen, Yujiong Zhu, Dingwei Dong, Yi Gui, Tao Zhang, Qi Huang, Xuanjing Artificial Intelligence In the maintenance of complex systems, fault trees are used to locate problems and provide targeted solutions. To enable fault trees stored as images to be directly processed by large language models, which can assist in tracking and analyzing malfunctions, we propose a novel textual representation of fault trees. Building on it, we construct a benchmark for multi-turn dialogue systems that emphasizes robust interaction in complex environments, evaluating a model's ability to assist in malfunction localization, which contains $3130$ entries and $40.75$ turns per entry on average. We train an end-to-end model to generate vague information to reflect user behavior and introduce long-range rollback and recovery procedures to simulate user error scenarios, enabling assessment of a model's integrated capabilities in task tracking and error recovery, and Gemini 2.5 pro archives the best performance. |
| title | JFTA-Bench: Evaluate LLM's Ability of Tracking and Analyzing Malfunctions Using Fault Trees |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2603.22978 |