Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.01259 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916725809741824 |
|---|---|
| author | Lichtenstein, Flavio de Souza, Daniel Alexandre Trufen, Carlos Eduardo Madureira Gonçalves, Victor Wendel da Silva Bernardes, Juliana de Paula Baroni, Vinicius Miranda DeOcesano-Pereira, Carlos Ormundo, Leonardo Fontoura de Souza, Fabio Augusto Labre Ibañez, Olga Celia Martinez Starobinas, Nancy Lopes, Luciano Rodrigo Fontes, Aparecida Maria de Andrade, Sonia Aparecida Chudzinski-Tavassi, Ana Marisa |
| author_facet | Lichtenstein, Flavio de Souza, Daniel Alexandre Trufen, Carlos Eduardo Madureira Gonçalves, Victor Wendel da Silva Bernardes, Juliana de Paula Baroni, Vinicius Miranda DeOcesano-Pereira, Carlos Ormundo, Leonardo Fontoura de Souza, Fabio Augusto Labre Ibañez, Olga Celia Martinez Starobinas, Nancy Lopes, Luciano Rodrigo Fontes, Aparecida Maria de Andrade, Sonia Aparecida Chudzinski-Tavassi, Ana Marisa |
| contents | A scientific study begins with a central question, and search engines like PubMed are the first tools for retrieving knowledge and understanding the current state of the art. Large Language Models (LLMs) have been used in research, promising acceleration and deeper results. However, besides caution, they demand rigorous validation. Assessing complex biological relationships remains challenging for SQL-based tools and LLM models. Here, we introduce the Digital Pathway Curation (DPC) pipeline to evaluate the reproducibility and accuracy of the Gemini models against PubMed search and human expert curation. Using two omics experiments, we created a large dataset (Ensemble) based on determining pathway-disease associations. With the Ensemble dataset, we demonstrate that Gemini achieves high run-to-run reproducibility of approximately 99% and inter-model reproducibility of around 75%. Next, we calculate the crowdsourced consensus using a smaller dataset. The CSC allows us to calculate accuracies, and the Gemini multi-model consensus reached a significant accuracy of about 87%. Our findings demonstrate that LLMs are reproducible, reliable, and valuable tools for navigating complex biomedical knowledge. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_01259 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Digital Pathway Curation (DPC): a pipeline able to assess the reproducibility, consensus and accuracy in biomedical search retrieval by comparing Gemini, PubMed, and Scientific Reviewers Lichtenstein, Flavio de Souza, Daniel Alexandre Trufen, Carlos Eduardo Madureira Gonçalves, Victor Wendel da Silva Bernardes, Juliana de Paula Baroni, Vinicius Miranda DeOcesano-Pereira, Carlos Ormundo, Leonardo Fontoura de Souza, Fabio Augusto Labre Ibañez, Olga Celia Martinez Starobinas, Nancy Lopes, Luciano Rodrigo Fontes, Aparecida Maria de Andrade, Sonia Aparecida Chudzinski-Tavassi, Ana Marisa Quantitative Methods I.2.0 A scientific study begins with a central question, and search engines like PubMed are the first tools for retrieving knowledge and understanding the current state of the art. Large Language Models (LLMs) have been used in research, promising acceleration and deeper results. However, besides caution, they demand rigorous validation. Assessing complex biological relationships remains challenging for SQL-based tools and LLM models. Here, we introduce the Digital Pathway Curation (DPC) pipeline to evaluate the reproducibility and accuracy of the Gemini models against PubMed search and human expert curation. Using two omics experiments, we created a large dataset (Ensemble) based on determining pathway-disease associations. With the Ensemble dataset, we demonstrate that Gemini achieves high run-to-run reproducibility of approximately 99% and inter-model reproducibility of around 75%. Next, we calculate the crowdsourced consensus using a smaller dataset. The CSC allows us to calculate accuracies, and the Gemini multi-model consensus reached a significant accuracy of about 87%. Our findings demonstrate that LLMs are reproducible, reliable, and valuable tools for navigating complex biomedical knowledge. |
| title | Digital Pathway Curation (DPC): a pipeline able to assess the reproducibility, consensus and accuracy in biomedical search retrieval by comparing Gemini, PubMed, and Scientific Reviewers |
| topic | Quantitative Methods I.2.0 |
| url | https://arxiv.org/abs/2505.01259 |