Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.13791 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910992808542208 |
|---|---|
| author | Rakotonirina, Nathanaël Carraz Hamdy, Mohammed Campos, Jon Ander Weber, Lucas Testoni, Alberto Fadaee, Marzieh Pezzelle, Sandro Del Tredici, Marco |
| author_facet | Rakotonirina, Nathanaël Carraz Hamdy, Mohammed Campos, Jon Ander Weber, Lucas Testoni, Alberto Fadaee, Marzieh Pezzelle, Sandro Del Tredici, Marco |
| contents | Large Language Models (LLMs) are increasingly used in working environments for a wide range of tasks, excelling at solving individual problems in isolation. However, are they also able to effectively collaborate over long-term interactions? To investigate this, we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_13791 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions Rakotonirina, Nathanaël Carraz Hamdy, Mohammed Campos, Jon Ander Weber, Lucas Testoni, Alberto Fadaee, Marzieh Pezzelle, Sandro Del Tredici, Marco Computation and Language Large Language Models (LLMs) are increasingly used in working environments for a wide range of tasks, excelling at solving individual problems in isolation. However, are they also able to effectively collaborate over long-term interactions? To investigate this, we introduce MemoryCode, a synthetic multi-session dataset designed to test LLMs' ability to track and execute simple coding instructions amid irrelevant information, simulating a realistic setting. While all the models we tested handle isolated instructions well, even the performance of state-of-the-art models like GPT-4o deteriorates when instructions are spread across sessions. Our analysis suggests this is due to their failure to retrieve and integrate information over long instruction chains. Our results highlight a fundamental limitation of current LLMs, restricting their ability to collaborate effectively in long interactions. |
| title | From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.13791 |