Do Code Models Suffer from the Dunning-Kruger Effect?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912632551768064 |
|---|---|
| author | Singh, Mukul Chatterjee, Somya Radhakrishna, Arjun Gulwani, Sumit |
| author_facet | Singh, Mukul Chatterjee, Somya Radhakrishna, Arjun Gulwani, Sumit |
| contents | As artificial intelligence systems increasingly collaborate with humans in creative and technical domains, questions arise about the cognitive boundaries and biases that shape our shared agency. This paper investigates the Dunning-Kruger Effect (DKE), the tendency for those with limited competence to overestimate their abilities in state-of-the-art LLMs in coding tasks. By analyzing model confidence and performance across a diverse set of programming languages, we reveal that AI models mirror human patterns of overconfidence, especially in unfamiliar or low-resource domains. Our experiments demonstrate that less competent models and those operating in rare programming languages exhibit stronger DKE-like bias, suggesting that the strength of the bias is proportionate to the competence of the models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05457 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Do Code Models Suffer from the Dunning-Kruger Effect? Singh, Mukul Chatterjee, Somya Radhakrishna, Arjun Gulwani, Sumit Artificial Intelligence Computation and Language Software Engineering As artificial intelligence systems increasingly collaborate with humans in creative and technical domains, questions arise about the cognitive boundaries and biases that shape our shared agency. This paper investigates the Dunning-Kruger Effect (DKE), the tendency for those with limited competence to overestimate their abilities in state-of-the-art LLMs in coding tasks. By analyzing model confidence and performance across a diverse set of programming languages, we reveal that AI models mirror human patterns of overconfidence, especially in unfamiliar or low-resource domains. Our experiments demonstrate that less competent models and those operating in rare programming languages exhibit stronger DKE-like bias, suggesting that the strength of the bias is proportionate to the competence of the models. |
| title | Do Code Models Suffer from the Dunning-Kruger Effect? |
| topic | Artificial Intelligence Computation and Language Software Engineering |
| url | https://arxiv.org/abs/2510.05457 |