Assessing large language models' accuracy in providing patient support for choroidal melanoma.

Anguita, Rodrigo; Downie, Catriona; Ferro Desideri, Lorenzo; Sagoo, Mandeep S (2024). Assessing large language models' accuracy in providing patient support for choroidal melanoma. (In Press). Eye Springer Nature 10.1038/s41433-024-03231-w

Text s41433-024-03231-w.pdf - Published Version Restricted to registered users only Available under License Publisher holds Copyright. Download (420kB) | Request a copy

PURPOSE

This study aimed to evaluate the accuracy of information that patients can obtain from large language models (LLMs) when seeking answers to common questions about choroidal melanoma.

METHODS

Comparative study comparing frequently asked questions from choroidal melanoma patients and queried three major LLMs-ChatGPT 3.5, Bing AI, and DocsGPT. Answers were reviewed by three ocular oncology experts and scored as accurate, partially accurate, or inaccurate. Statistical analysis compared the quality of responses across models.

RESULTS

For medical advice questions, ChatGPT gave 92% accurate responses compared to 58% for Bing AI and DocsGPT. For pre/post-op questions, ChatGPT and Bing AI were 86% accurate while DocsGPT was 73% accurate. There were no statistically significant differences between models. ChatGPT responses were the longest while Bing AI responses were the shortest, but length did not affect accuracy. All LLMs appropriately directed patients to seek medical advice from professionals.

CONCLUSION

LLMs show promising capability to address common choroidal melanoma patient questions at generally acceptable accuracy levels. However, inconsistent, and inaccurate responses do occur, highlighting the need for improved fine-tuning and oversight before integration into clinical practice.

Interest & Impact

Downloads

0 since deposited on 15 Jul 2024
0 in the past 12 months

Citations

5 Citations in Web of Science ®
6 Citations in Scopus

Search

in Google Scholar™

Services

Actions (login required)

Edit item

Actions (login required)

Edit item