Anguita, Rodrigo; Downie, Catriona; Ferro Desideri, Lorenzo; Sagoo, Mandeep S (2024). Assessing large language models' accuracy in providing patient support for choroidal melanoma. (In Press). Eye Springer Nature 10.1038/s41433-024-03231-w
![[img]](https://boris.unibe.ch/style/images/fileicons/text.png) |
Text
s41433-024-03231-w.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (420kB) |
PURPOSE
This study aimed to evaluate the accuracy of information that patients can obtain from large language models (LLMs) when seeking answers to common questions about choroidal melanoma.
METHODS
Comparative study comparing frequently asked questions from choroidal melanoma patients and queried three major LLMs-ChatGPT 3.5, Bing AI, and DocsGPT. Answers were reviewed by three ocular oncology experts and scored as accurate, partially accurate, or inaccurate. Statistical analysis compared the quality of responses across models.
RESULTS
For medical advice questions, ChatGPT gave 92% accurate responses compared to 58% for Bing AI and DocsGPT. For pre/post-op questions, ChatGPT and Bing AI were 86% accurate while DocsGPT was 73% accurate. There were no statistically significant differences between models. ChatGPT responses were the longest while Bing AI responses were the shortest, but length did not affect accuracy. All LLMs appropriately directed patients to seek medical advice from professionals.
CONCLUSION
LLMs show promising capability to address common choroidal melanoma patient questions at generally acceptable accuracy levels. However, inconsistent, and inaccurate responses do occur, highlighting the need for improved fine-tuning and oversight before integration into clinical practice.
Item Type: |
Journal Article (Original Article) |
|---|---|
Division/Institute: |
04 Faculty of Medicine > Department of Head Organs and Neurology (DKNS) > Clinic of Ophthalmology |
UniBE Contributor: |
Anguita Henríquez, Rodrigo Andrés, Ferro Desideri, Lorenzo |
Subjects: |
600 Technology > 610 Medicine & health |
ISSN: |
1476-5454 |
Publisher: |
Springer Nature |
Language: |
English |
Submitter: |
Pubmed Import |
Date Deposited: |
15 Jul 2024 10:43 |
Last Modified: |
15 Jul 2024 10:52 |
Publisher DOI: |
10.1038/s41433-024-03231-w |
PubMed ID: |
39003430 |
BORIS DOI: |
10.48350/198999 |
URI: |
https://boris.unibe.ch/id/eprint/198999 |
Actions (login required)
 |
Edit item |