Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917707531681792 |
|---|---|
| author | Nikandrou, Malvina Pantazopoulos, Georgios Konstas, Ioannis Suglia, Alessandro |
| author_facet | Nikandrou, Malvina Pantazopoulos, Georgios Konstas, Ioannis Suglia, Alessandro |
| contents | Continual learning focuses on incrementally training a model on a sequence of tasks with the aim of learning new tasks while minimizing performance drop on previous tasks. Existing approaches at the intersection of Continual Learning and Visual Question Answering (VQA) do not study how the multimodal nature of the input affects the learning dynamics of a model. In this paper, we demonstrate that each modality evolves at different rates across a continuum of tasks and that this behavior occurs in established encoder-only models as well as modern recipes for developing Vision & Language (VL) models. Motivated by this observation, we propose a modality-aware feature distillation (MAFED) approach which outperforms existing baselines across models of varying scale in three multimodal continual learning settings. Furthermore, we provide ablations showcasing that modality-aware distillation complements experience replay. Overall, our results emphasize the importance of addressing modality-specific dynamics to prevent forgetting in multimodal continual learning. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19297 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation Nikandrou, Malvina Pantazopoulos, Georgios Konstas, Ioannis Suglia, Alessandro Computer Vision and Pattern Recognition Continual learning focuses on incrementally training a model on a sequence of tasks with the aim of learning new tasks while minimizing performance drop on previous tasks. Existing approaches at the intersection of Continual Learning and Visual Question Answering (VQA) do not study how the multimodal nature of the input affects the learning dynamics of a model. In this paper, we demonstrate that each modality evolves at different rates across a continuum of tasks and that this behavior occurs in established encoder-only models as well as modern recipes for developing Vision & Language (VL) models. Motivated by this observation, we propose a modality-aware feature distillation (MAFED) approach which outperforms existing baselines across models of varying scale in three multimodal continual learning settings. Furthermore, we provide ablations showcasing that modality-aware distillation complements experience replay. Overall, our results emphasize the importance of addressing modality-specific dynamics to prevent forgetting in multimodal continual learning. |
| title | Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.19297 |