Exploring State Tracking Capabilities of Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915615782993920 |
|---|---|
| author | Rezaee, Kiamehr Camacho-Collados, Jose Pilehvar, Mohammad Taher |
| author_facet | Rezaee, Kiamehr Camacho-Collados, Jose Pilehvar, Mohammad Taher |
| contents | Large Language Models (LLMs) have demonstrated impressive capabilities in solving complex tasks, including those requiring a certain level of reasoning. In this paper, we focus on state tracking, a problem where models need to keep track of the state governing a number of entities. To isolate the state tracking component from other factors, we propose a benchmark based on three well-defined state tracking tasks and analyse the performance of LLMs in different scenarios. The results indicate that the recent generation of LLMs (specifically, GPT-4 and Llama3) are capable of tracking state, especially when integrated with mechanisms such as Chain of Thought. However, models from the former generation, while understanding the task and being able to solve it at the initial stages, often fail at this task after a certain number of steps. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_10457 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring State Tracking Capabilities of Large Language Models Rezaee, Kiamehr Camacho-Collados, Jose Pilehvar, Mohammad Taher Computation and Language 68T50 I.2.7 Large Language Models (LLMs) have demonstrated impressive capabilities in solving complex tasks, including those requiring a certain level of reasoning. In this paper, we focus on state tracking, a problem where models need to keep track of the state governing a number of entities. To isolate the state tracking component from other factors, we propose a benchmark based on three well-defined state tracking tasks and analyse the performance of LLMs in different scenarios. The results indicate that the recent generation of LLMs (specifically, GPT-4 and Llama3) are capable of tracking state, especially when integrated with mechanisms such as Chain of Thought. However, models from the former generation, while understanding the task and being able to solve it at the initial stages, often fail at this task after a certain number of steps. |
| title | Exploring State Tracking Capabilities of Large Language Models |
| topic | Computation and Language 68T50 I.2.7 |
| url | https://arxiv.org/abs/2511.10457 |