Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908468662763520 |
|---|---|
| author | Truyts, Cesar Augusto Madid Rabelo, Amanda Gomes de Souza, Gabriel Mesquita Lages, Daniel Scaldaferri Pereira, Adriano Jose Flato, Uri Adrian Prync Reis, Eduardo Pontes dos Vieira, Joaquim Edson Silveira, Paulo Sergio Panse Junior, Edson Amaro |
| author_facet | Truyts, Cesar Augusto Madid Rabelo, Amanda Gomes de Souza, Gabriel Mesquita Lages, Daniel Scaldaferri Pereira, Adriano Jose Flato, Uri Adrian Prync Reis, Eduardo Pontes dos Vieira, Joaquim Edson Silveira, Paulo Sergio Panse Junior, Edson Amaro |
| contents | Artificial intelligence (AI) has shown the potential to revolutionize healthcare by improving diagnostic accuracy, optimizing workflows, and personalizing treatment plans. Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have achieved notable advancements in natural language processing and medical applications. However, the evaluation of these models has focused predominantly on the English language, leading to potential biases in their performance across different languages.
This study investigates the capability of six LLMs (GPT-4.0 Turbo, LLaMA-3-8B, LLaMA-3-70B, Mixtral 8x7B Instruct, Titan Text G1-Express, and Command R+) and four MLLMs (Claude-3.5-Sonnet, Claude-3-Opus, Claude-3-Sonnet, and Claude-3-Haiku) to answer questions written in Brazilian spoken portuguese from the medical residency entrance exam of the Hospital das Clínicas da Faculdade de Medicina da Universidade de São Paulo (HCFMUSP) - the largest health complex in South America. The performance of the models was benchmarked against human candidates, analyzing accuracy, processing time, and coherence of the generated explanations.
The results show that while some models, particularly Claude-3.5-Sonnet and Claude-3-Opus, achieved accuracy levels comparable to human candidates, performance gaps persist, particularly in multimodal questions requiring image interpretation. Furthermore, the study highlights language disparities, emphasizing the need for further fine-tuning and data set augmentation for non-English medical AI applications.
Our findings reinforce the importance of evaluating generative AI in various linguistic and clinical settings to ensure a fair and reliable deployment in healthcare. Future research should explore improved training methodologies, improved multimodal reasoning, and real-world clinical integration of AI-driven medical assistance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_19885 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam Truyts, Cesar Augusto Madid Rabelo, Amanda Gomes de Souza, Gabriel Mesquita Lages, Daniel Scaldaferri Pereira, Adriano Jose Flato, Uri Adrian Prync Reis, Eduardo Pontes dos Vieira, Joaquim Edson Silveira, Paulo Sergio Panse Junior, Edson Amaro Computation and Language Artificial intelligence (AI) has shown the potential to revolutionize healthcare by improving diagnostic accuracy, optimizing workflows, and personalizing treatment plans. Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) have achieved notable advancements in natural language processing and medical applications. However, the evaluation of these models has focused predominantly on the English language, leading to potential biases in their performance across different languages. This study investigates the capability of six LLMs (GPT-4.0 Turbo, LLaMA-3-8B, LLaMA-3-70B, Mixtral 8x7B Instruct, Titan Text G1-Express, and Command R+) and four MLLMs (Claude-3.5-Sonnet, Claude-3-Opus, Claude-3-Sonnet, and Claude-3-Haiku) to answer questions written in Brazilian spoken portuguese from the medical residency entrance exam of the Hospital das Clínicas da Faculdade de Medicina da Universidade de São Paulo (HCFMUSP) - the largest health complex in South America. The performance of the models was benchmarked against human candidates, analyzing accuracy, processing time, and coherence of the generated explanations. The results show that while some models, particularly Claude-3.5-Sonnet and Claude-3-Opus, achieved accuracy levels comparable to human candidates, performance gaps persist, particularly in multimodal questions requiring image interpretation. Furthermore, the study highlights language disparities, emphasizing the need for further fine-tuning and data set augmentation for non-English medical AI applications. Our findings reinforce the importance of evaluating generative AI in various linguistic and clinical settings to ensure a fair and reliable deployment in healthcare. Future research should explore improved training methodologies, improved multimodal reasoning, and real-world clinical integration of AI-driven medical assistance. |
| title | Zero-shot Performance of Generative AI in Brazilian Portuguese Medical Exam |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.19885 |