WatChat: Explaining perplexing programs by debugging mental models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909331978452992 |
|---|---|
| author | Chandra, Kartik Collins, Katherine M. Crichton, Will Chen, Tony Li, Tzu-Mao Weller, Adrian Nigam, Rachit Tenenbaum, Joshua Ragan-Kelley, Jonathan |
| author_facet | Chandra, Kartik Collins, Katherine M. Crichton, Will Chen, Tony Li, Tzu-Mao Weller, Adrian Nigam, Rachit Tenenbaum, Joshua Ragan-Kelley, Jonathan |
| contents | Often, a good explanation for a program's unexpected behavior is a bug in the programmer's code. But sometimes, an even better explanation is a bug in the programmer's mental model of the language or API they are using. Instead of merely debugging our current code ("giving the programmer a fish"), what if our tools could directly debug our mental models ("teaching the programmer to fish")? In this paper, we apply recent ideas from computational cognitive science to offer a principled framework for doing exactly that. Given a "why?" question about a program, we automatically infer potential misconceptions about the language/API that might cause the user to be surprised by the program's behavior -- and then analyze those misconceptions to provide explanations of the program's behavior. Our key idea is to formally represent misconceptions as counterfactual (erroneous) semantics for the language/API, which can be inferred and debugged using program synthesis techniques. We demonstrate our framework, WatChat, by building systems for explanation in two domains: JavaScript type coercion, and the Git version control system. We evaluate WatChatJS and WatChatGit by comparing their outputs to experimentally-collected human-written explanations in these two domains: we show that WatChat's explanations exhibit key features of human-written explanation, unlike those of a state-of-the-art language model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_05334 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WatChat: Explaining perplexing programs by debugging mental models Chandra, Kartik Collins, Katherine M. Crichton, Will Chen, Tony Li, Tzu-Mao Weller, Adrian Nigam, Rachit Tenenbaum, Joshua Ragan-Kelley, Jonathan Programming Languages Artificial Intelligence Human-Computer Interaction Often, a good explanation for a program's unexpected behavior is a bug in the programmer's code. But sometimes, an even better explanation is a bug in the programmer's mental model of the language or API they are using. Instead of merely debugging our current code ("giving the programmer a fish"), what if our tools could directly debug our mental models ("teaching the programmer to fish")? In this paper, we apply recent ideas from computational cognitive science to offer a principled framework for doing exactly that. Given a "why?" question about a program, we automatically infer potential misconceptions about the language/API that might cause the user to be surprised by the program's behavior -- and then analyze those misconceptions to provide explanations of the program's behavior. Our key idea is to formally represent misconceptions as counterfactual (erroneous) semantics for the language/API, which can be inferred and debugged using program synthesis techniques. We demonstrate our framework, WatChat, by building systems for explanation in two domains: JavaScript type coercion, and the Git version control system. We evaluate WatChatJS and WatChatGit by comparing their outputs to experimentally-collected human-written explanations in these two domains: we show that WatChat's explanations exhibit key features of human-written explanation, unlike those of a state-of-the-art language model. |
| title | WatChat: Explaining perplexing programs by debugging mental models |
| topic | Programming Languages Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2403.05334 |