Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917946907951104 |
|---|---|
| author | Jiang, Jiyue Chen, Pengan Wang, Jiuming He, Dongchen Wei, Ziqin Hong, Liang Zong, Licheng Wang, Sheng Yu, Qinze Ma, Zixian Chen, Yanyu Fan, Yimin Shi, Xiangyu Sun, Jiawei Wu, Chuan Li, Yu |
| author_facet | Jiang, Jiyue Chen, Pengan Wang, Jiuming He, Dongchen Wei, Ziqin Hong, Liang Zong, Licheng Wang, Sheng Yu, Qinze Ma, Zixian Chen, Yanyu Fan, Yimin Shi, Xiangyu Sun, Jiawei Wu, Chuan Li, Yu |
| contents | Large language models (LLMs) have become important tools in solving biological problems, offering improvements in accuracy and adaptability over conventional methods. Several benchmarks have been proposed to evaluate the performance of these LLMs. However, current benchmarks can hardly evaluate the performance of these models across diverse tasks effectively. In this paper, we introduce a comprehensive prompting-based benchmarking framework, termed Bio-benchmark, which includes 30 key bioinformatics tasks covering areas such as proteins, RNA, drugs, electronic health records, and traditional Chinese medicine. Using this benchmark, we evaluate six mainstream LLMs, including GPT-4o and Llama-3.1-70b, etc., using 0-shot and few-shot Chain-of-Thought (CoT) settings without fine-tuning to reveal their intrinsic capabilities. To improve the efficiency of our evaluations, we demonstrate BioFinder, a new tool for extracting answers from LLM responses, which increases extraction accuracy by round 30% compared to existing methods. Our benchmark results show the biological tasks suitable for current LLMs and identify specific areas requiring enhancement. Furthermore, we propose targeted prompt engineering strategies for optimizing LLM performance in these contexts. Based on these findings, we provide recommendations for the development of more robust LLMs tailored for various biological applications. This work offers a comprehensive evaluation framework and robust tools to support the application of LLMs in bioinformatics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04013 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting Jiang, Jiyue Chen, Pengan Wang, Jiuming He, Dongchen Wei, Ziqin Hong, Liang Zong, Licheng Wang, Sheng Yu, Qinze Ma, Zixian Chen, Yanyu Fan, Yimin Shi, Xiangyu Sun, Jiawei Wu, Chuan Li, Yu Computation and Language Artificial Intelligence Large language models (LLMs) have become important tools in solving biological problems, offering improvements in accuracy and adaptability over conventional methods. Several benchmarks have been proposed to evaluate the performance of these LLMs. However, current benchmarks can hardly evaluate the performance of these models across diverse tasks effectively. In this paper, we introduce a comprehensive prompting-based benchmarking framework, termed Bio-benchmark, which includes 30 key bioinformatics tasks covering areas such as proteins, RNA, drugs, electronic health records, and traditional Chinese medicine. Using this benchmark, we evaluate six mainstream LLMs, including GPT-4o and Llama-3.1-70b, etc., using 0-shot and few-shot Chain-of-Thought (CoT) settings without fine-tuning to reveal their intrinsic capabilities. To improve the efficiency of our evaluations, we demonstrate BioFinder, a new tool for extracting answers from LLM responses, which increases extraction accuracy by round 30% compared to existing methods. Our benchmark results show the biological tasks suitable for current LLMs and identify specific areas requiring enhancement. Furthermore, we propose targeted prompt engineering strategies for optimizing LLM performance in these contexts. Based on these findings, we provide recommendations for the development of more robust LLMs tailored for various biological applications. This work offers a comprehensive evaluation framework and robust tools to support the application of LLMs in bioinformatics. |
| title | Benchmarking Large Language Models on Multiple Tasks in Bioinformatics NLP with Prompting |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2503.04013 |