Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911456970145792 |
|---|---|
| author | Maharaj, Kishan Menon, Nandakishore Saxena, Ashita Tamilselvam, Srikanth |
| author_facet | Maharaj, Kishan Menon, Nandakishore Saxena, Ashita Tamilselvam, Srikanth |
| contents | Large language models (LLMs) increasingly assist software engineering tasks that require reasoning over long code contexts, yet their robustness under varying input conditions remains unclear. We conduct a systematic study of long-context code question answering using controlled ablations that test sensitivity to answer format, distractors, and context scale. Extending LongCodeBench Python dataset with new COBOL and Java question-answer sets, we evaluate state-of-the-art models under three settings: (i) shuffled multiple-choice options, (ii) open-ended questions and (iii) needle-in-a-haystack contexts containing relevant and adversarially irrelevant information. Results show substantial performance drops in both shuffled multiple-choice options and open-ended questions, and brittle behavior in the presence of irrelevant cues. Our findings highlight limitations of current long-context evaluations and provide a broader benchmark for assessing code reasoning in both legacy and modern systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_17183 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering Maharaj, Kishan Menon, Nandakishore Saxena, Ashita Tamilselvam, Srikanth Software Engineering Artificial Intelligence Large language models (LLMs) increasingly assist software engineering tasks that require reasoning over long code contexts, yet their robustness under varying input conditions remains unclear. We conduct a systematic study of long-context code question answering using controlled ablations that test sensitivity to answer format, distractors, and context scale. Extending LongCodeBench Python dataset with new COBOL and Java question-answer sets, we evaluate state-of-the-art models under three settings: (i) shuffled multiple-choice options, (ii) open-ended questions and (iii) needle-in-a-haystack contexts containing relevant and adversarially irrelevant information. Results show substantial performance drops in both shuffled multiple-choice options and open-ended questions, and brittle behavior in the presence of irrelevant cues. Our findings highlight limitations of current long-context evaluations and provide a broader benchmark for assessing code reasoning in both legacy and modern systems. |
| title | Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2602.17183 |