Evaluation of large language models for discovery of gene set function
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929609050685440 |
|---|---|
| author | Hu, Mengzhou Alkhairy, Sahar Lee, Ingoo Pillich, Rudolf T. Fong, Dylan Smith, Kevin Bachelder, Robin Ideker, Trey Pratt, Dexter |
| author_facet | Hu, Mengzhou Alkhairy, Sahar Lee, Ingoo Pillich, Rudolf T. Fong, Dylan Smith, Kevin Bachelder, Robin Ideker, Trey Pratt, Dexter |
| contents | Gene set analysis is a mainstay of functional genomics, but it relies on curated databases of gene functions that are incomplete. Here we evaluate five Large Language Models (LLMs) for their ability to discover the common biological functions represented by a gene set, substantiated by supporting rationale, citations and a confidence assessment. Benchmarking against canonical gene sets from the Gene Ontology, GPT-4 confidently recovered the curated name or a more general concept (73% of cases), while benchmarking against random gene sets correctly yielded zero confidence. Gemini-Pro and Mixtral-Instruct showed ability in naming but were falsely confident for random sets, whereas Llama2-70b had poor performance overall. In gene sets derived from 'omics data, GPT-4 identified novel functions not reported by classical functional enrichment (32% of cases), which independent review indicated were largely verifiable and not hallucinations. The ability to rapidly synthesize common gene functions positions LLMs as valuable 'omics assistants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_04019 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Evaluation of large language models for discovery of gene set function Hu, Mengzhou Alkhairy, Sahar Lee, Ingoo Pillich, Rudolf T. Fong, Dylan Smith, Kevin Bachelder, Robin Ideker, Trey Pratt, Dexter Genomics Artificial Intelligence Computation and Language Molecular Networks Gene set analysis is a mainstay of functional genomics, but it relies on curated databases of gene functions that are incomplete. Here we evaluate five Large Language Models (LLMs) for their ability to discover the common biological functions represented by a gene set, substantiated by supporting rationale, citations and a confidence assessment. Benchmarking against canonical gene sets from the Gene Ontology, GPT-4 confidently recovered the curated name or a more general concept (73% of cases), while benchmarking against random gene sets correctly yielded zero confidence. Gemini-Pro and Mixtral-Instruct showed ability in naming but were falsely confident for random sets, whereas Llama2-70b had poor performance overall. In gene sets derived from 'omics data, GPT-4 identified novel functions not reported by classical functional enrichment (32% of cases), which independent review indicated were largely verifiable and not hallucinations. The ability to rapidly synthesize common gene functions positions LLMs as valuable 'omics assistants. |
| title | Evaluation of large language models for discovery of gene set function |
| topic | Genomics Artificial Intelligence Computation and Language Molecular Networks |
| url | https://arxiv.org/abs/2309.04019 |