Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ
Fuente:
arXiv
Guardado en:
| Autores principales: | , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918301625483264 |
|---|---|
| author | Neergaard, Karl Qiu, Le Chersoni, Emmanuele |
| author_facet | Neergaard, Karl Qiu, Le Chersoni, Emmanuele |
| contents | Single-prompt evaluations dominate current LLM benchmarking, yet they fail to capture the conversational dynamics where real-world harm occurs. In this study, we examined whether conversation length affects response veracity by evaluating LLM performance on the BoolQ dataset under varying length and scaffolding conditions. Our results across three distinct LLMs revealed model-specific vulnerabilities that are invisible under single-turn testing. The length-dependent and scaffold-specific effects we observed demonstrate a fundamental limitation of static evaluations, as deployment-relevant vulnerabilities could only be spotted in a multi-turn conversational setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_16508 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ Neergaard, Karl Qiu, Le Chersoni, Emmanuele Computation and Language Single-prompt evaluations dominate current LLM benchmarking, yet they fail to capture the conversational dynamics where real-world harm occurs. In this study, we examined whether conversation length affects response veracity by evaluating LLM performance on the BoolQ dataset under varying length and scaffolding conditions. Our results across three distinct LLMs revealed model-specific vulnerabilities that are invisible under single-turn testing. The length-dependent and scaffold-specific effects we observed demonstrate a fundamental limitation of static evaluations, as deployment-relevant vulnerabilities could only be spotted in a multi-turn conversational setting. |
| title | Is Length Really A Liability? An Evaluation of Multi-turn LLM Conversations using BoolQ |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.16508 |