Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2406.06474 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929380476846080 |
|---|---|
| author | Cosentino, Justin Belyaeva, Anastasiya Liu, Xin Furlotte, Nicholas A. Yang, Zhun Lee, Chace Schenck, Erik Patel, Yojan Cui, Jian Schneider, Logan Douglas Bryant, Robby Gomes, Ryan G. Jiang, Allen Lee, Roy Liu, Yun Perez, Javier Rogers, Jameson K. Speed, Cathy Tailor, Shyam Walker, Megan Yu, Jeffrey Althoff, Tim Heneghan, Conor Hernandez, John Malhotra, Mark Stern, Leor Matias, Yossi Corrado, Greg S. Patel, Shwetak Shetty, Shravya Zhan, Jiening Prabhakara, Shruthi McDuff, Daniel McLean, Cory Y. |
| author_facet | Cosentino, Justin Belyaeva, Anastasiya Liu, Xin Furlotte, Nicholas A. Yang, Zhun Lee, Chace Schenck, Erik Patel, Yojan Cui, Jian Schneider, Logan Douglas Bryant, Robby Gomes, Ryan G. Jiang, Allen Lee, Roy Liu, Yun Perez, Javier Rogers, Jameson K. Speed, Cathy Tailor, Shyam Walker, Megan Yu, Jeffrey Althoff, Tim Heneghan, Conor Hernandez, John Malhotra, Mark Stern, Leor Matias, Yossi Corrado, Greg S. Patel, Shwetak Shetty, Shravya Zhan, Jiening Prabhakara, Shruthi McDuff, Daniel McLean, Cory Y. |
| contents | In health, most large language model (LLM) research has focused on clinical tasks. However, mobile and wearable devices, which are rarely integrated into such tasks, provide rich, longitudinal data for personal health monitoring. Here we present Personal Health Large Language Model (PH-LLM), fine-tuned from Gemini for understanding and reasoning over numerical time-series personal health data. We created and curated three datasets that test 1) production of personalized insights and recommendations from sleep patterns, physical activity, and physiological responses, 2) expert domain knowledge, and 3) prediction of self-reported sleep outcomes. For the first task we designed 857 case studies in collaboration with domain experts to assess real-world scenarios in sleep and fitness. Through comprehensive evaluation of domain-specific rubrics, we observed that Gemini Ultra 1.0 and PH-LLM are not statistically different from expert performance in fitness and, while experts remain superior for sleep, fine-tuning PH-LLM provided significant improvements in using relevant domain knowledge and personalizing information for sleep insights. We evaluated PH-LLM domain knowledge using multiple choice sleep medicine and fitness examinations. PH-LLM achieved 79% on sleep and 88% on fitness, exceeding average scores from a sample of human experts. Finally, we trained PH-LLM to predict self-reported sleep quality outcomes from textual and multimodal encoding representations of wearable data, and demonstrate that multimodal encoding is required to match performance of specialized discriminative models. Although further development and evaluation are necessary in the safety-critical personal health domain, these results demonstrate both the broad knowledge and capabilities of Gemini models and the benefit of contextualizing physiological data for personal health applications as done with PH-LLM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06474 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards a Personal Health Large Language Model Cosentino, Justin Belyaeva, Anastasiya Liu, Xin Furlotte, Nicholas A. Yang, Zhun Lee, Chace Schenck, Erik Patel, Yojan Cui, Jian Schneider, Logan Douglas Bryant, Robby Gomes, Ryan G. Jiang, Allen Lee, Roy Liu, Yun Perez, Javier Rogers, Jameson K. Speed, Cathy Tailor, Shyam Walker, Megan Yu, Jeffrey Althoff, Tim Heneghan, Conor Hernandez, John Malhotra, Mark Stern, Leor Matias, Yossi Corrado, Greg S. Patel, Shwetak Shetty, Shravya Zhan, Jiening Prabhakara, Shruthi McDuff, Daniel McLean, Cory Y. Artificial Intelligence Computation and Language In health, most large language model (LLM) research has focused on clinical tasks. However, mobile and wearable devices, which are rarely integrated into such tasks, provide rich, longitudinal data for personal health monitoring. Here we present Personal Health Large Language Model (PH-LLM), fine-tuned from Gemini for understanding and reasoning over numerical time-series personal health data. We created and curated three datasets that test 1) production of personalized insights and recommendations from sleep patterns, physical activity, and physiological responses, 2) expert domain knowledge, and 3) prediction of self-reported sleep outcomes. For the first task we designed 857 case studies in collaboration with domain experts to assess real-world scenarios in sleep and fitness. Through comprehensive evaluation of domain-specific rubrics, we observed that Gemini Ultra 1.0 and PH-LLM are not statistically different from expert performance in fitness and, while experts remain superior for sleep, fine-tuning PH-LLM provided significant improvements in using relevant domain knowledge and personalizing information for sleep insights. We evaluated PH-LLM domain knowledge using multiple choice sleep medicine and fitness examinations. PH-LLM achieved 79% on sleep and 88% on fitness, exceeding average scores from a sample of human experts. Finally, we trained PH-LLM to predict self-reported sleep quality outcomes from textual and multimodal encoding representations of wearable data, and demonstrate that multimodal encoding is required to match performance of specialized discriminative models. Although further development and evaluation are necessary in the safety-critical personal health domain, these results demonstrate both the broad knowledge and capabilities of Gemini models and the benefit of contextualizing physiological data for personal health applications as done with PH-LLM. |
| title | Towards a Personal Health Large Language Model |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2406.06474 |