Privacy-Aware Document Visual Question Answering
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910587767750656 |
|---|---|
| author | Tito, Rubèn Nguyen, Khanh Tobaben, Marlon Kerkouche, Raouf Souibgui, Mohamed Ali Jung, Kangsoo Jälkö, Joonas D'Andecy, Vincent Poulain Joseph, Aurelie Kang, Lei Valveny, Ernest Honkela, Antti Fritz, Mario Karatzas, Dimosthenis |
| author_facet | Tito, Rubèn Nguyen, Khanh Tobaben, Marlon Kerkouche, Raouf Souibgui, Mohamed Ali Jung, Kangsoo Jälkö, Joonas D'Andecy, Vincent Poulain Joseph, Aurelie Kang, Lei Valveny, Ernest Honkela, Antti Fritz, Mario Karatzas, Dimosthenis |
| contents | Document Visual Question Answering (DocVQA) has quickly grown into a central task of document understanding. But despite the fact that documents contain sensitive or copyrighted information, none of the current DocVQA methods offers strong privacy guarantees. In this work, we explore privacy in the domain of DocVQA for the first time, highlighting privacy issues in state of the art multi-modal LLM models used for DocVQA, and explore possible solutions. Specifically, we focus on invoice processing as a realistic document understanding scenario, and propose a large scale DocVQA dataset comprising invoice documents and associated questions and answers. We employ a federated learning scheme, that reflects the real-life distribution of documents in different businesses, and we explore the use case where the data of the invoice provider is the sensitive information to be protected. We demonstrate that non-private models tend to memorise, a behaviour that can lead to exposing private information. We then evaluate baseline training schemes employing federated learning and differential privacy in this multi-modal scenario, where the sensitive information might be exposed through either or both of the two input modalities: vision (document image) or language (OCR tokens). Finally, we design attacks exploiting the memorisation effect of the model, and demonstrate their effectiveness in probing a representative DocVQA models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_10108 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Privacy-Aware Document Visual Question Answering Tito, Rubèn Nguyen, Khanh Tobaben, Marlon Kerkouche, Raouf Souibgui, Mohamed Ali Jung, Kangsoo Jälkö, Joonas D'Andecy, Vincent Poulain Joseph, Aurelie Kang, Lei Valveny, Ernest Honkela, Antti Fritz, Mario Karatzas, Dimosthenis Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning Document Visual Question Answering (DocVQA) has quickly grown into a central task of document understanding. But despite the fact that documents contain sensitive or copyrighted information, none of the current DocVQA methods offers strong privacy guarantees. In this work, we explore privacy in the domain of DocVQA for the first time, highlighting privacy issues in state of the art multi-modal LLM models used for DocVQA, and explore possible solutions. Specifically, we focus on invoice processing as a realistic document understanding scenario, and propose a large scale DocVQA dataset comprising invoice documents and associated questions and answers. We employ a federated learning scheme, that reflects the real-life distribution of documents in different businesses, and we explore the use case where the data of the invoice provider is the sensitive information to be protected. We demonstrate that non-private models tend to memorise, a behaviour that can lead to exposing private information. We then evaluate baseline training schemes employing federated learning and differential privacy in this multi-modal scenario, where the sensitive information might be exposed through either or both of the two input modalities: vision (document image) or language (OCR tokens). Finally, we design attacks exploiting the memorisation effect of the model, and demonstrate their effectiveness in probing a representative DocVQA models. |
| title | Privacy-Aware Document Visual Question Answering |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2312.10108 |