Learning Interpretable Queries for Explainable Image Classification with Information Pursuit
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915427299360768 |
|---|---|
| author | Kolek, Stefan Chattopadhyay, Aditya Chan, Kwan Ho Ryan Andrade-Loarca, Hector Kutyniok, Gitta Vidal, Réne |
| author_facet | Kolek, Stefan Chattopadhyay, Aditya Chan, Kwan Ho Ryan Andrade-Loarca, Hector Kutyniok, Gitta Vidal, Réne |
| contents | Information Pursuit (IP) is an explainable prediction algorithm that greedily selects a sequence of interpretable queries about the data in order of information gain, updating its posterior at each step based on observed query-answer pairs. The standard paradigm uses hand-crafted dictionaries of potential data queries curated by a domain expert or a large language model after a human prompt. However, in practice, hand-crafted dictionaries are limited by the expertise of the curator and the heuristics of prompt engineering. This paper introduces a novel approach: learning a dictionary of interpretable queries directly from the dataset. Our query dictionary learning problem is formulated as an optimization problem by augmenting IP's variational formulation with learnable dictionary parameters. To formulate learnable and interpretable queries, we leverage the latent space of large vision and language models like CLIP. To solve the optimization problem, we propose a new query dictionary learning algorithm inspired by classical sparse dictionary learning. Our experiments demonstrate that learned dictionaries significantly outperform hand-crafted dictionaries generated with large language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_11548 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Learning Interpretable Queries for Explainable Image Classification with Information Pursuit Kolek, Stefan Chattopadhyay, Aditya Chan, Kwan Ho Ryan Andrade-Loarca, Hector Kutyniok, Gitta Vidal, Réne Computer Vision and Pattern Recognition Information Pursuit (IP) is an explainable prediction algorithm that greedily selects a sequence of interpretable queries about the data in order of information gain, updating its posterior at each step based on observed query-answer pairs. The standard paradigm uses hand-crafted dictionaries of potential data queries curated by a domain expert or a large language model after a human prompt. However, in practice, hand-crafted dictionaries are limited by the expertise of the curator and the heuristics of prompt engineering. This paper introduces a novel approach: learning a dictionary of interpretable queries directly from the dataset. Our query dictionary learning problem is formulated as an optimization problem by augmenting IP's variational formulation with learnable dictionary parameters. To formulate learnable and interpretable queries, we leverage the latent space of large vision and language models like CLIP. To solve the optimization problem, we propose a new query dictionary learning algorithm inspired by classical sparse dictionary learning. Our experiments demonstrate that learned dictionaries significantly outperform hand-crafted dictionaries generated with large language models. |
| title | Learning Interpretable Queries for Explainable Image Classification with Information Pursuit |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2312.11548 |